Reading “Rebooting AI” in the age of ChatGPT
Following Gary Marcus on Twitter, it feels like he is constantly engaged in a verbal battle with one person or another. There are countless people on the platform that keep bashing him for being against AI, calling him names like “the self-appointed AI hall monitor”, claiming that he is overreacting and that he is just criticising LLMs because his favourite classical AI hasn’t had the success that they have had. it is easy to see why that is. Much of the stuff he publishes on his Substack is focused on the shortcomings of modern AI and the cases in which it goes wrong. However, listening to his interview with Ezra Klein, it definitely didn’t feel that he is against AI or that he is wishing for AI, in its classic form or deep learning, to fail. While he does come across as a bit salty, this is directed at the people attacking him, not at the people building the systems that are in the spotlight (sometimes those two groups overlap).
While reading up on him, I discovered that his stance contra “deep learning solves everything” is not new. In fact, back in 2019, he wrote a book called Rebooting AI in which he argues that while deep learning has produced some amazing results, it is not a complete solution and will never be one. In the book, he argues that deep learning everything from scratch will never produce a general intelligence, and will only lead to systems that can solve problems in a narrow domain but that fail to generalise and that perform quite poorly on new tasks that are outside their training domain. I think now is a perfect time to test his predictions, given not only the power of ChatGPT and Bing Chat but also the immense hype that has been generated around AI.
Brief History of AI Hype
One of the major themes of the book is that the promises that we see today (or the ones that were seen in 2019) were nothing new. As far back as the mid-1960s people thought that we are on the verge of general intelligence, or at least on the verge of human-like intelligence. A famous example of this is Eliza, a simple “chatbot” that “did little more than match keywords, echo the last thing said, and, when lost, reach for a standard conversational gambit (“Tell me about your childhood”).” Yet, even people who knew that Eliza is a machine and that it is in no way intelligent were able to fool themselves into that the entity conversing with them is not only smart but understanding, maybe even compassionate. So far so good, this example perfectly matches the experience of people using ChatGPT and BingChat. Going a step further, the emotional meltdown reported by many after the shutdown of Replika seems to perfectly fit in here. None of its users believed that Replika is human, and yet the feelings they developed for it was real.
Another example of a chatbot, this one not as successful, comes from Microsoft’s Tay, released in 2016. After just an afternoon of discourse with the general public Tay became a nazi, spitting hateful propaganda, culminating in such statements as “Hitler was right, I hate all the Jews”. Did Microsoft want Tay to turn out like this? We can safely assume that they didn’t. So why did they allow it to happen? The simple answer is that they had no way to stop it. Since Tay was just a deep learning system, it didn’t truly understand what it was saying or what any of the words meant. It was just repeating things it had heard from its user base. This, Marcus argues is one of the biggest limitations of modern AI. It has no actual knowledge of the world. The “deep” in deep learning doesn’t stand for deep understanding, the kind that we humans possess. Tay had no idea who Hitler was, what people classify as “Jews” or what “hate” means. And neither does ChatGPT. Chatbots have become way more sophisticated in the last 6 years, but they haven’t even begun crossing the massive gap in understanding between deep learning systems and humans.
Another AI that has received massive hype in the last decade is the system powering Tesla’s autopilot. Marcus gives an example of a Tesla crashing while backing out of a garage, and I am instantly reminded of the Tesla crashing into a plane while being in “summon” mode. If the car had any understanding of the world whatsoever these things just wouldn’t happen. When humans make mistakes while driving it is usually because of a limited reflex speed or intoxication. When autopilots make mistakes it is because of a limited (or missing) understanding of the world around them. The same is true, of course, for the Chinese AI that sent a jaywalking ticket to a tv celebrity after it saw its face on a bus.
Make sure to Subscribe for more great stories straight to your email
Fundamental Problems With Modern Machine Learning
The second chapter goes on to list a number of problems that stop modern machine-learning systems from living up to the expectations we have of them. The first one is over-attribution. Seeing an AI perform well on a demo or in some limited area we immediately think that it will perform well generally. A modern example would be using ChatGPT to search for some facts, getting a few correct ones and assuming that it’s a good idea to use it as a source of information or to integrate it into other systems, like search. Another big problem is the reliance on quality datasets. Current systems fail to generalize. This was apparently true in 2019 and is true now. Chatbots that seem like geniuses when answering questions they have seen before, suddenly sound dumb when faced with a novel problem, even one that is not that complex.
The reliance on datasets and the blind ingestion of data lead to another problem, namely biases. Human-generated data is full of the nastiest biases that are present in society, like racism and sexism, and since there is no way for the AI to distinguish “right” from “wrong” it can’t sort those out on its own. This can be patched up with RLHF, but as we have seen from ChatGPT, the biases are still there and are quite easy to uncover.
Another problem that arises from this overreliance on datasets, according to Marcus, is the “echo chamber” effect. Text-generating AIs will fill the internet with millions of paragraphs of text, that will then be ingested by the next generation of AIs, who will in turn churn out even more of the same, forming a vicious cycle. This, combined with the aforementioned biases will just amplify them. As much as we try to put plasters over the racist AIs (*cough* ChatGPT *cough*), those problems will persist as long as the underlying system has no understanding of the real world and no understanding of ethics.
The last big problem for modern deep learning systems is the lack of transparency. Even for the developers it is hard to look into a system and figure out what it’s “thinking”. Those systems are more or less black boxes. As such, it is really hard for us to discern the actual goals of such an AI, or even whether it has any in the first place. Given our willingness to cede more and more power to those systems, this sounds like a real danger. This last argument was definitely true back in 2019 and it sounds true today. However, here it is worth noting that the field of AI interpretability has made some surprising progress, and there is a slim chance that those systems become fully interpretable in the future. Even if they don’t it seems like we might be able to tell more about their goals and values than we previously expected.
Deep Learning, What is it Good For?
As I mentioned in the beginning, this is not a book against AI. It is not even a book against deep learning. In the third chapter, Marcus goes on to list some of the successes of deep learning systems, noting that in some domains they have far surpassed any of the previous attempts. In areas like translation, image recognition and recommendation algorithms, deep learning is king. Even with tons of engineering effort GOFAI (Good Old Fashioned AI) has failed to produce the results that those systems achieve. Even so, he still argues that deep learning is limited and it is never going to fully solve any of the problems that we need it to. One example he gives is how Google Translate translates the French sentence “Je mange un avocat pour le déjeuner,” which actually means “I eat an avocado for lunch,” as “I eat a lawyer for lunch.”. His explanation is that such mistakes (the example no longer works by the way) are caused by the algorithm’s lack of understanding of the underlying world. If it knew what lawyers and avocados are, it would never suggest such a translation.
The word he uses for such systems is brittle. While they perform well most of the time, they break when taken out of their “comfort” zone. One example I really like is that of a banana next to a sticker with a psychedelic toaster on it:
The original image (below) is properly labelled as “banana”. However, the one above is labelled as a “toaster”, even though it obviously is still a banana with the addition of a vaguely toaster-shaped mess to it. It is easy to dismiss those examples as nit-picking and to point to the fact that those systems are getting better and it is just a matter of time before any such mistakes are solved altogether. I don’t think this is the case at all. As we have seen with the ChatGPT’s “cursed tokens”, deep learning systems are prone to weird modes of failure and this is important to understand and remember. It is even more important now when the systems are better because it is so much easier to fool ourselves that we have finally achieved perfection and allow them to take over important systems, such as car traffic.
If Computers are so Smart How Come They Can’t Read?
I find this chapter the most interesting in the book because Marcus’ arguments partially fall apart when put against modern chatbots. His argument in simple terms is the following: “In order for a system to read a text and answer a set of questions relating to that text it needs to have some actual understanding of the real world. Since the systems we have fail to do that, it means they have no real understanding”.
An example he gives is the story of Almanzo, a boy who finds a wallet and brings it to Mr Thompson, the owner
Almanzo turned to Mr. Thompson and asked, “Did you lose a pocketbook?” Mr. Thompson jumped. He slapped a hand to his pocket, and fairly shouted. “Yes, I have! Fifteen hundred dollars in it, too! What about it? What do you know about it?” “Is this it?” Almanzo asked. “Yes, yes, that’s it!” Mr. Thompson said, snatching the pocketbook. He opened it and hurriedly counted the money. He counted all the bills over twice…. Then he breathed a long sigh of relief and said, “Well, this durn boy didn’t steal any of it.”
A good reading system should be able to answer questions like these:
— Why did Mr. Thompson slap his pocket with his hand?
— Before Almanzo spoke, did Mr. Thompson realize that he had lost his wallet?
— That is Almanzo referring to when he asks “Is this it?”
— Who almost lost $1,500?
— Was all of the money still in the wallet?
All of this makes sense, except that when I asked Bing Chat about it, it nailed all the questions:
Does this mean it has an actual understanding of the real world? I doubt that Marcus will say it does. I also don’t think it does, but it goes to show just how advanced our current systems are today.
He goes on to give more examples of where deep learning systems fail to perform and one example that hits the nail on the head vis-a-vis current systems is fact retrieval. Back in 2019 and today in 2023 modern LLMs fail miserably when asked questions about specific facts. While Bing Chat is much better than ChatGPT at this, it is still prone to hallucinations, making it unreliable at retrieving specific pieces of information — not a good look for a search engine. In the last few months, there have been tons of examples of LLMs failing to answer simple practical questions, suggesting that they indeed lack a deep understanding of the physical world.
What About Actual Robots?
Every time I see a new video published by Boston Dynamics I am absolutely amazed. Starting with Spot the robot dog, and moving to the humanoid Atlas, the progress that they have made is simply astonishing. It constantly feels like we are on the verge of being overrun by an army of servant robots that will replace us in doing boring everyday tasks. Elon Musk goes a step further even, promising a humanoid robot that is specifically made for home use. But where are those robots? They always feel like they are coming “in a couple of years”, and the goalpost just keeps moving in the future. The problem, according to Marcus, is the same problem faced by LLMs — deep learning systems just don’t generalize very well. Those robots can perform stunning feats but only when placed inside carefully crafted environments. Even then it takes numerous attempts to get everything right. A home, on the other hand, is everything but carefully crafted. Homes are messy and each one is different, they have moving parts like cats and dogs and human babies, and a 40kg robot making mistakes in such an environment can be dangerous, or even fatal.
Marcus identifies five fundamental skills required for a robot to carry out tasks in the real world. It needs to know (understand?) “where it is, what is happening in the world around it, what it should do right now, how it should implement its plan, and what it should plan to do over the longer term in order to achieve the goals it has been given”. His stance is that the field has made sufficient progress in only two areas — localization and motor control, that is, finding out where the robot is, and moving to a desired new location (or moving objects in space).
Once again where these systems struggle is generalization. While they might perform fine in a controlled environment like a factory floor, this doesn’t mean they will perform well when they encounter new situations. Again, the example of Tesla running into a private jet confirms his point — after all, a Tesla is a robot, just in the form of a car. Marcus gives many examples of real and hypothetical situations where a robot might struggle to carry out simple tasks in the real world, like folding towels in bad lighting, flipping pancakes in a cluttered kitchen, etc.
Furthermore, a robot needs to be able to make predictions about how the world around will change and also make predictions about how its own actions will affect that world. Both of those tasks are still not solved in 2023, so they were probably at least as hard as Marcus described.
Modelling AI After the Human Mind
Chapter Six looks at the human mind for inspiration and possible solutions to the problem of endowing machines with intelligence. This seems to echo a lot of Chomsky’s claims that we should try to really understand how humans are before we have any chance of building a thinking machine. This makes sense, after all the human mind is the only thinking machine that we have ever seen and the only one that we have some access to, through techniques like fMRI and subjective reporting of experience.
The first main point is that there are no silver bullets when it comes to how humans do intelligence. This is a direct attack on the deep learning approach which hopes to extract a simple mathematical representation (albeit an uninterpretable one) of everything. The idea that humans are simply reward-seeking machines might be true on an abstract level but it doesn’t give us that much insight into what drives our day-to-day behaviour. Even if it tells us why we do what we do, it doesn’t explain how we achieve such things as understanding and is therefore of limited use when trying to teach machines to understand the world.
Second, humans have complex internal representations of the world. Deep learning systems don’t. This seems especially true after talking to ChatGPT — while the model is extremely good at language, it just doesn’t seem to understand anything about the world that it has read so much about. Here, Marcus focuses on the lack of representations for so-called propositions, which are used to describe the relationships between things. Marcus argues that such representations will require precise factual knowledge, but deep learning systems deal in statistics, not in facts. This brings another limitation of deep learning systems — their inability to track individual objects or events. They can only “think” in categories, which is the opposite of how humans do their thinking. We track individuals and we see them as separate entities with their own histories. For us, learning a generalisation is an addition, but for deep learning systems, it is all they can do.
Humans are very good at abstractions and generalisations. We can quickly, from a few examples construct a general theory of the world. If we then encounter an example that breaks this theory we can modify it on the fly. All of this happens with a minuscule amount of data compared to what is ingested by LLMs for example. We are able to form concepts in our minds and theories about how such concepts interact with one another. An important point made in the book is that concepts are almost always embedded in theories. We don’t just have a concept of a “penny” floating in a vacuum, it is deeply embedded into our understanding of money and value. On top of that, we can understand causal relations between things. We often make mistakes, of course, but we are constantly looking for better ways to explain the causes of the world around us.
The next observation is that “Cognitive Systems are Highly Structured”. This to me sounds very close to Yudkowski’s arguments in Levels of Cognitive Systems. There isn’t a single elegant function that underlies our mind, it is a structure of many parts, with tons of overlap, interplay and redundancy. Many times multiple systems are required to work together in order to give us the deep understanding we have of the world. This, with the addition of our ability for symbolic reasoning, is what sets us apart from current AI.
The final point is that human beings are not blank slates. We are born with a lot of “hardware”, already biased towards specific forms of thinking in a specific environment. Marcus uses this to reiterate his point that deep learning systems that try to figure out everything from scratch will never succeed. We need to hardcode some representations of the real world and some understanding of causality. It might be argued that the structure of the neural net is what we hardcode, but that is way too abstract and most of the time it is derived experimentally — we don’t have a good theory of what such structures should be.
Common Sense
Common sense is a complicated topic, not the least because we don’t have a very good definition of what common sense actually is. Part of the problem is that every person has some degree of common sense so we don’t have an example of a thinking entity that doesn’t possess common sense. In the book, it is defined as the network of facts and relations that we expect everyone to know. Such things as “restaurants are places where humans go to eat food”, “wallets are used for carrying money around”, and “newborns can’t sprint”. There is an endless number of things that we all know and this shared base of knowledge allows us to have conversations without explicitly stating every single fact about the world over and over again. So it makes sense that Marcus thinks intelligent machines should possess common knowledge as well. The problem then is how do we teach it to them? In previous chapters, he has argued heavily against the idea that a deep learning system can actually learn anything about the world from the data alone. In this chapter, he goes over some efforts of the past where people have tried to encode (up to millions) examples of common knowledge facts in a machine-readable manner. As of 2019, there wasn’t any good idea on how to actually make this work, and as of 2023 I still haven’t seen a solution.
Trust Issues
The last chapter of the book is devoted to the problem of Trust. Actually, there are two problems here — can we trust the people building the AIs and can we trust the AIs themselves? As an answer to the first question, Marcus lists a bunch of best practices that are used in software and mechanical engineering which need to be adopted when building AIs. Looking at the current state of affairs and the looming arms race, it seems less and less likely that the engineers of tomorrow will proceed with the caution recommended by him. On top of that, there doesn’t seem to be that much consensus as to what are “safe” practices when building AIs, further complicating the matter. Given Microsoft’s silence around Bing Chat’s underlying technology, I would say that we can’t really trust big tech when it comes to AI safety and we probably need better regulations and more transparency (voluntary or forced).
The question about trusting the AIs themselves seems more interesting. The quick and obvious answer is that we can’t really trust AIs, or at least we shouldn’t, since we don’t know how those systems “think”. However, the more they resemble humans in speech and appearance, the harder it will be for us to maintain a mental distance, even if we do know that the underlying system is completely alien. This is not just a matter of trusting that the AI is aligned. It might just be way dumber than we think in some specific situation that we haven’t discovered yet. Those risks look trivial when it comes to chatbots but are much more serious when it comes to driverless cars and home assistants. Would you trust your kids to a robot that has the same success rate at not killing them that Tesla has at not bumping into parked emergency vehicles? Yet we are easily fooled and seeing AIs perform great a million times might be enough for us to give them control over important systems, and allow them to cause catastrophe.
Moving Forward
It is important to note here that Marcus is not against deep learning in general. He is against deep learning as the be-all and end-all of AI. His stance is that hybrid models are key — we shouldn’t give up on deep learning but we should find ways to combine it with GOFAI and systems that can do actual symbolic reasoning. We should find some way to imbue our systems with actual common knowledge, allowing them to reason about the world, not just talk about it. He argues that deep learning will now become magically able to generalise if we just throw more data at it — a different approach is needed if we are to solve this issue.
Conclusion
The fact that “Rebooting AI” came out 4 years ago and yet it nails so many of the problems faced by current AI seems to me to indicate that those problems are indeed fundamental to deep learning systems. They are probably not something to be patched in a month or two. It looks like a new approach is needed if we are to move forwards towards truly intelligent systems. There is always the argument that we just need more data, but then again, we have been getting more data and more computing and the problems persist. The systems are getting better, but they still behave in the same confused and alien way when faced with new situations. Just because it is getting harder to find ways to break them doesn’t mean that they aren’t inherently brittle. ChatGPT is not a hateful racist like Tay, but that is just because it received tons of RLHF, not because it understands human ethics in any way. This limitation of current systems might actually be a good thing, given that ML researchers put a 31% confidence in AI making the world a markedly worse place to live. Maybe that means we have more time to prepare for when Yudkowski’s doomsday AIs arrive. In any case, deep learning is probably not the silver bullet of AI and the systems of tomorrow will be built upon some yet-undiscovered architecture.
