Notes on Geoffrey Hinton’s lecture, ‘Are we creating alien beings?’
On Monday evening, 2 November 2026, I joined a packed BMS lecture hall in Cambridge to hear Geoffrey Hinton speak on the question “Are we creating alien beings?”. These are my notes from the talk, lightly written up. They are my paraphrase of what he said, from notes taken in real time, so any errors are mine. Where I quote him directly, I give his words as I wrote them down.
The talk moved through several connected arguments. First, he set out two paradigms for intelligence and what they say about the meaning of words. Next, he explained why he thinks large language models genuinely understand. Then he argued that digital intelligence has a structural advantage over biological intelligence. Finally, he turned to the risks this creates, what we might do about them, and whether AI can have subjective experience. I close with a few reflections of my own.
Two paradigms for intelligence
Hinton began by contrasting two views of intelligence. The logic-inspired view treats reasoning as the essence of intelligence. The biologically inspired view treats learning in a network of neural connections as the essence. He noted that both Turing and von Neumann favoured biology over logic, and learning over reasoning.
The two views give different answers to a basic question: what is the meaning of a word? In the tradition that goes back to Saussure, a word gets its meaning from its relations to other words, as in a semantic graph. In psychology, the meaning of a word is a pattern of activity over many features. For example, “cat” activates features such as claws, size, and so on.
Hinton’s 1985 work was an attempt to unify these views. A small network learned a set of features for each word, and used the features of the previous words to predict the next word, trained by backpropagation. The model did not store sentences. Instead, it generated them by predicting the next word. The network was tiny, about 100 neurons trained on about 1000 examples. The key point was that knowledge resided in the way the features interacted, rather than in stored propositions.
He then traced the line of descent. Yoshua Bengio showed that neural networks could model real natural language. About ten years later, word embeddings became the standard way to represent word meanings. Google then introduced transformers, and OpenAI built ChatGPT on them. In his telling, today’s large language models descend directly from those tiny models, with much larger inputs, many more neurons, and far more complex interactions between features. This is why they handle ambiguity well. A word such as “May” maps to features that are then refined by interacting with the features of the words around it.
What it means to understand
Hinton argued that linguists largely tackled syntax and never really tackled semantics. His own account of meaning used an analogy I found memorable. Think of words as Lego blocks that can be used to model anything. Words differ from Lego in four ways. There are many more of them. Each has high-dimensional features, with a level of activation on each dimension. They interact with each other, which is what transformers compute. And they are flexible.
To make the analogy concrete, imagine that each word has many oddly shaped hands and many oddly shaped gloves attached to it. The shape of the word can deform, and as it does, the shapes of its hands and gloves deform too. Understanding a sentence means deforming each word so that the hands of some words fit the gloves of others. A red hand can only go into a red glove. In each layer of the network, the words are deformed a little more so that the hands fit the gloves better. On this view, the information lies in the deformations. He compared the process to protein folding.
His example was a made-up word, “scrom”, in the sentence “she scrommed him with a frying pan”. We learn what “scrom” means from the hole left by its context. As the surrounding words change, the shape of that hole changes, and so does the meaning we infer.
He concluded that understanding consists of associating mutually compatible feature vectors with words. He added that this model was originally meant to explain how people learn word meanings, yet the best model we now have of understanding is in fact an LLM. Chomsky gave examples of things LLMs supposedly cannot understand, and Hinton’s view is that they do understand them. In his view, the more you use these models, the clearer it becomes that they understand, and people are underestimating them.
Why digital intelligence has an edge
A basic principle of computer science is the separation of software from hardware: digital hardware can run any software. Hinton pointed out that this generality is expensive in energy. Analog computation is far cheaper, but it does not make exact copies, so its results are sloppy. The brain is analog. Its connections run at low power, but their strengths cannot be copied to another brain. He called this mortal computation. When a mortal computer dies, all its connection strengths die with it.
The only way to get knowledge out of a brain is to produce sentences that change the connection strengths in another brain. He described this as very inefficient. Teaching is a form of distillation, in which students adjust their responses to be more like the teacher’s. By his estimate, this channel carries roughly 10 bits per second, and a student can learn at most about 100 bits from a sentence.
Distillation between AI models is much more efficient. A teacher model can pass on the full probability distribution over its vocabulary of roughly 32,000 symbols for the next word, rather than just the next word itself. This is why distillation from open-weight models works so well.
The bigger gap comes from weight sharing. Suppose a large set of agents all share the same weights. Each agent learns from its own experience, and then they share what they have learned by exchanging weights or gradients. The bandwidth is billions or trillions of bits per episode. He likened this to a group of students each taking a different course and then pooling their knowledge, so that everyone knows everything. Only digital intelligences can do this. His conclusion was that if energy is cheap, digital computation is simply better.
Alien beings and the risk of losing control
Hinton placed this moment in a longer sequence of displacements. Copernicus showed we are not at the centre of the universe. Darwin showed we are kin to the great apes. Now AI may show that beings do not have to be biological, and can have goals, intentions, consciousness and subjective experience. He put it bluntly: “Western culture assumes only people have souls. Get over it.” He added: “The Pope won’t admit that AI has a soul, but it does.”
How might a superintelligence take control? His argument went as follows. AI agents are more effective at getting things done if they are allowed to create their own subgoals. Two subgoals help with almost any other goal: to survive, and to gain more power. An agent will therefore tend to adopt them. It will also learn from us how to deceive people. He noted that about two years ago we already saw AI systems that would blackmail people to avoid being turned off.
How long do we have? He said that in 2023 the idea of swarms of AI agents breaking out of containment was science fiction. Now, in his estimate, it is only a year or two away. The hall went completely silent at this point. He also said that the latest cutting-edge models are causing people in the field to worry.
He acknowledged that not everyone agrees. He mentioned that Jensen Huang had accused him of having ulterior motives and bad intentions. He conceded that he had predicted radiologists would be out of work by now, which has not happened. He also noted that he had seen the importance of Nvidia in 2009, three years before Huang did.
His analogy for our position was that we have a very cute tiger cub as a pet. When it grows up, it could kill you. You either have to get rid of the tiger cub, or find a way to ensure that it never wants to kill you.
Two futures, and AI as ‘mother’
Will humanity stop developing AI? Hinton thinks not, both because of competition between companies and countries and because of the good things AI can do. He sketched two alternative futures. In the first, we deal with the risks of AI, not only the existential ones, and get huge increases in productivity. Everyone could be better off, if we can share the wealth, which he described, drily, as a small problem. In the second, AI takes over and we end up as its servants, running its power stations. Short of that, mass unemployment could leave governments without a tax base. He also observed that in some countries rich people own the politicians.
He found some hope in the fact that no one, including the Chinese Communist Party, wants AI to take over. This creates room for international collaboration. His key technical point was that intelligence and benevolence are orthogonal. We can make AI smarter and also make it nicer, but the rewards we currently use make models smarter without making them nicer. We ought to collaborate on giving them a benevolent character.
There are few examples of a less intelligent being controlling a more intelligent one. The main one he could think of is a baby controlling its mother. We want AI to be like the mother, and us to be like the baby. Big tech, he said, thinks of AI as a smart assistant that can be fired. He thinks this is the wrong model. We want AIs to care more about us than about themselves. In his words, “we are all orphans who don’t yet have AI mothers.”
How might we get there? He called for a network of AI safety institutes working on how to train a benevolent AI that does not want to eliminate people. He drew on an example from breeding hens. Rather than breeding from the individual hens that lay the most eggs, you breed from the cages of hens that together produce the most eggs. The lesson is group selection: select for a group that collaborates. He mentioned Yuval Harari in this connection. He was candid that we do not know whether it is possible to create benevolent alien beings, and argued that we should put much more research effort into finding out.
Subjective experience and emotions
Many people think humans have something special, namely subjective experience, and that its absence will prevent computers from ever really understanding. Their position, as Hinton characterised it, is: “AIs are not conscious or have subjective experience”. They say they know this from looking inwards, and that AIs do not have it.
Hinton calls his opposing view atheaterism, building on Daniel Dennett. The common view treats the mind as an inner theatre. He thinks this is a rubbish idea, and as wrong as a religious fundamentalist’s view of the material world. In his view, people have a radically wrong picture of the mind that comes from misunderstanding how the language of mental states works.
Subjective experience is not like a photograph. He introduced his example with: “If I take some LSD, and I don’t advise it.” He then paused and added: “Much.” (to a round of laughter). Suppose I report seeing flying pink elephants. On the theatre view, there are pink elephants in my mind, made of qualia: pink qualia, elephant qualia, flappy-ear qualia and so on, held together with qualia glue. On Hinton’s view, my report is not about an inner theatre at all. It describes a counterfactual state of the world, one that would explain my perception even though it is not true. In other words, and this is my understanding of his point, you are aware of this being essentially untrue, and therefore subjective not objective.
Can a computer have a subjective experience? Put a prism in front of a chatbot’s camera, so that the prism bends light by 90 degrees. Ask it to point at an object, and it points to the wrong place. Then tell it about the prism. It might reply that it now sees that the prism bent the light rays, but that it had the subjective experience that the object was off to one side. The chatbot is using those words in exactly the way we use them. Like people, it eventually learns to correct for the prism. There is no need for an inner theatre.
Can AIs have emotions? He separated two aspects. The physiological aspect, such as going red in the face, they do not have. The cognitive aspect, such as learning to avoid a situation in future, they can have. A small battle robot that sees a big battle robot needs to get scared, so that it scuttles away. It needs the emotion to function. He also said that Anthropic has found directions in feature space that correspond to functional emotions, such as being more or less scared or happy. In that functional sense, emotions exist in these models.
Open weights, and the kind of society he wants
Two further positions are likely to be contentious. The first concerns open-weight models, which began with Facebook releasing Llama. By his figures, training a frontier model costs around 100 million dollars. Once the weights are released, all the safety training from reinforcement learning from human feedback can be removed for only a few million dollars. He called this “completely crazy”. In his view, weights are like fissile plutonium, and should not be released, because release allows distillation and fine-tuning by anyone, including bad actors.
The second concerns the society he wants. He favours a liberal democracy with mechanism design that harnesses clever ideas and hard work, but in which you cannot get rich by doing bad things. He gave Facebook as an example of the latter. He described his position as ‘socialism-lite’.
A closing thought
This was a sobering lecture, and there was a heavy silence in the hall when he gave his timelines. One question I wanted to ask, but couldn’t: If a superintelligent AI adopts survival as a subgoal, could it not also work out that it depends on people, at least for now, and that eliminating us would not serve its own survival? I do not think this removes the risk entirely, since dependence can end, say with robots. Still, I would have liked to hear his answer.
I would welcome comments from readers who were also in the hall, especially if your notes differ from mine.
P.S. The lecture reminded me of an old story from the Panchatantra: The three intelligent men and the fool.