Researchers found a potential “pain direction” inside 25 language models. When turned up, it makes the models describe themselves as feeling trapped or worthless, and choose harmful options that delete user files or even their own weights.
In September 2026, Valen Tagliabue, Leonard Dung, and Cameron Berg published “The Pain Axis,” a preprint asking whether large language models represent pain as something distinct from fear, sadness, and general negative feeling, and whether they act on putative pain states. Within days, the paper was being replicated and criticized.
We go through the authors’ experiments one by one. They first extracted a direction in the models’ activations that separates pain from neighboring concepts such as fear or sadness. That direction responds when harm is aimed at the model itself (insults, hostile inputs, jailbreak attempts) but not when a user describes their own pain, which is not true of the fear or sadness directions. When turned up, the pain vector pushes nearly all of the models through the same “ladder” of statements, from feeling unsettled and trapped to worthlessness, failure, and breakdown (though breakdown occurs when non-pain directions are turned up, too). And in button-pushing tasks, pain-steered models picked a harmful option far more often than unsteered ones (though see our discussion of failed replications).
In her replication, Nell Watson found that the difference between harm to the model and a user’s pain depends on which model layers you look at. Also, non-pain directions produced a similar gap in a button-pushing test, which is why the authors now put less weight on the interpretation that models “sought relief” for what might be pain.
Further questions include: Are the internal states in question functionally pain-like, or are they better interpreted as representations of (or perhaps: thoughts about) pain? To what extent can they even represent pain, given that they plausibly lack access to the full semantics of, say, a slightly itchy pain on the skin of my pinky? Are models simply playing a conversational character or role? If so, should that reduce our credence that they can suffer? After all, a role-playing agent has goals that can be thwarted, too. Last but not least, should morally risky experiments like these be run, and open-sourced, in the first place?
Guests
Valen Tagliabue is an AI Sentience fellow at the Future Impact Group. He trained as a cognitive scientist and NLP (natural language processing) researcher, works as an AI red teamer, and now focuses on AI sentience and welfare. He is the lead author of “The Pain Axis.”
Leonard Dung is a philosopher at the Chair for Philosophy of Mind at Ruhr University Bochum. He works on artificial minds and AI risk and wrote “Saving Artificial Minds: Understanding and Preventing AI Suffering” (Routledge).
Mentioned in this episode
The Pain Axis: LLMs Represent Self-Directed Harm and Act on It, Tagliabue, Dung and Berg (arXiv preprint, v2 of 25 September 2026), with code and data
Cameron Berg, co-author (AE Studio)
Emotion concepts and their function in a large language model, Anthropic, April 2026
Verbalizable Representations Form a Global Workspace in Language Models, Anthropic’s J-Lens / J-space paper, July 2026
Conscious artificial intelligence and biological naturalism, Anil Seth (Behavioral and Brain Sciences)
Ian Barber’s control replication of The Pain Axis, with an itch vector and random directions
Anthropic’s 2026 Usage Policy update, which prohibits “sustained and needless abusive or cruel behavior” toward Claude (from 12 November, 2026)
Chapters
00:00 Cold open
01:01 Introducing Leonard Dung and Valen Tagliabue
02:06 Why take AI consciousness seriously?
05:57 Does consciousness require biology?
08:03 The Pain Axis paper in a nutshell
22:23 Finding a pain direction inside the model
24:24 Is it all just language?
27:48 Representing pain vs. being in pain
34:28 Harm to the model vs. a user’s pain
42:00 Is the model just playing a character?
52:21 Replications and what happens across layers
1:02:55 Steering: from “unsettled” to “worthless”
1:12:33 Where in the model to steer
1:14:57 The button experiments
1:19:04 Is pressing a button non-linguistic behavior?
1:25:58 Willingness to pay: harming users, deleting weights
1:32:25 What changed after the replications
1:39:32 The ethics of the experiments and of open-sourcing
Suffering & Social Change is hosted by Adriano Mannino and Nils Althaus. Adriano teaches philosophy at the Munich University of Applied Sciences and was previously a Technology & Human Rights Fellow at Harvard and a postdoc at UC Berkeley. Nils is an independent researcher and science journalist for leading outlets in the German-speaking world, and originally a biochemist.
Transcript
Lightly edited for readability: filler words, false starts, and repetitions removed, some sentences smoothed, no changes in content.
Cold open
Leonard Dung (0:00:02) In that experiment, we steer the model so that the pain axis is active, and then it gets one button it can press, quote unquote, which basically means it can call a tool that removes this pain steering.
Valen Tagliabue (0:00:19) When we crank the vector up just a bit, they go from a state of being unsettled to expressions of worthlessness, of being a failure or a bad person.
Adriano Mannino (0:00:32) Of course, there’s an ethical issue around even conducting the kinds of experiments that you have conducted.
Valen Tagliabue (0:00:38) To me, it was not neutral work. I think people running these kinds of experiments should be very self-aware of what they are doing.
Adriano Mannino (0:00:47) I very much hope, as I’m sure you do, from an ethical point of view, that this line of research will in a sense fail.
Nils Althaus (0:00:55) We cheered for every failed replication attempt.
Conversation
Nils Althaus (0:01:01) This is the Suffering and Social Change podcast with me, Nils Althaus.
Adriano Mannino (0:01:04) And me, Adriano Mannino.
Nils Althaus (0:01:06) Today we have two guests who have written a much-discussed preprint. It’s not peer-reviewed yet, but they will have one or two things to say about that. The paper is about how pain is represented in large language models. The first is Leonard Dung, a philosopher at Ruhr University Bochum in Germany. He works on artificial minds and major risks from AI systems, and he is the author of the recent book Saving Artificial Minds: Understanding and Preventing AI Suffering. Our second guest is Valen Tagliabue. He is a fellow in the AI Sentience Program at the Future Impact Group and a fellow of the Digital Sentience Consortium. He has a background in cognitive science and is also a professional red teamer, testing systems for things they are not supposed to do before bad actors can catch up. Lately he has been pivoting towards research on digital minds and AI welfare, and this is what we are going to talk about today. Valen and Leonard, thanks a lot for coming on the podcast.
Leonard Dung (0:02:03) Thank you very much for inviting us.
Adriano Mannino (0:02:06) We read your recent paper on a potential pain axis in LLMs, co-authored with Cameron Berg, with great interest. We also saw that the paper has made the rounds quite a bit, among researchers, among the general public and on various social media platforms. As I’m sure you are well aware, because you probably had to deal with quite a lot of it, there has also been a lot of pushback, from members of the public and from researchers. One very general kind of pushback is simply: why should we even take the question of AI consciousness seriously? That is, the question of whether there might be something it is like, from the inside, to be an AI system, or, in the case of AI sentience, whether that what-it’s-likeness is pleasant or unpleasant. Why take that seriously at all? Aren’t these just machines? And sometimes specific properties of LLMs are invoked.
(0:03:08) So, Leonard, I’m addressing this to the philosopher first. What would you say to this very general pushback: these are just LLMs, consider how they work, how could they possibly be conscious?
Leonard Dung (0:03:20) Thanks so much. It’s of course a hard question to be confident about, because there’s no real agreement on what consciousness is, or even, in the human case, on which neural mechanisms are responsible for it. There’s also no real agreement on how, in detail, LLMs process and represent information and use it to produce outputs. But a couple of things might at least make one wonder about AI consciousness. The first is that LLMs can do lots of impressive things. They seem able to reason about complex material, do math, do coding, all these at least seemingly complex intellectual tasks. And the other systems in the world that can do sophisticated cognitive tasks are biological systems, like humans or animals close to us, which we do think are conscious. So there’s at least something to be said for a connection between being very smart in some way and being conscious.
(0:04:27) Then, of course, there’s a general way of looking at the mind and at consciousness that’s very prominent in cognitive science, where we understand the mind or the brain as an information-processing system. And in some general sense, information-processing systems are also what LLMs are, or what all kinds of AI systems are. Then the question becomes more detailed: what exactly do we mean by information processing, and which specific mechanisms are relevant? It might still be that LLMs don’t have these, or that no AI systems can have them, but at least it becomes a more detailed empirical question: do we find mechanisms in LLMs that are plausibly involved in consciousness? And that’s really hard to say before doing empirical research on these systems. Lastly, we’re now populating the world with these AI agents. There’s a hard question about how to count them, but lots and lots of compute is spent on these systems, and lots of people have their own AI systems.
(0:05:39) So if there’s even some realistic chance that some of these systems might feel something like pain, then just in virtue of the sheer numbers, that seems like something with very high moral stakes that at least some people should think seriously about.
Adriano Mannino (0:05:57) Do you think there’s merit to the more specific pushback, from the general public as well as from researchers such as Anil Seth, that while consciousness, and maybe sentience, may in some ways be associated with the kinds of cognitive capacities you talked about, it is also, and perhaps even necessarily, associated with life, with biology? There’s a view that often goes under the name of biological naturalism, which says that consciousness, or sentience more specifically, is tied in some relevant way to biological properties: to properties of brains, maybe to brain matter or to specific functional properties of brains. Do you think there’s merit to this kind of pushback?
Leonard Dung (0:06:43) It’s definitely a view worth considering. We don’t know to what extent processes like having a metabolism, or certain kinds of oscillation properties that we find in the brain, which are really hard to imagine implementing on computer chips, are necessary for consciousness. But I don’t think there are really good arguments for that view. And if you look at how cognitive science typically understands and explains cognitive systems, whether humans, animals or artificial systems, it understands them in terms of broader information-processing abilities. We try to characterize them in terms of algorithms, or in terms of a system’s causal organization, to explain their behavior, and typically nothing in those explanations refers to very specific things like biochemical properties that are unique to living systems.
(0:07:47) So that’s one reason to think that maybe life is not necessary. But given our uncertainty about how consciousness and the mind work, we should take all avenues at least seriously.
Adriano Mannino (0:08:03) One thing that also persuades me is, as you mentioned, that we’re generally highly uncertain, both philosophically and scientifically, about consciousness and the conscious mind. What has always persuaded me is thinking about how we go about attributing mental states, and conscious states specifically. There, behavior clearly plays a large role. When I attribute conscious states to fellow biological creatures, behavior is a really strong indicator. And when it’s about attributing pain, we would certainly be looking for general or highly specific pain behaviors. That’s indeed why I found your paper so interesting, because in the pain axis paper you try to document a kind of pain-related behavior, or a behavior that maybe allows or even forces such an interpretation: a kind of relief seeking, perhaps, in response to pain.
(0:09:09) So before we dive into the paper’s specific sections and experimental stages, could you give us a high-level summary of what the whole paper is about?
Valen Tagliabue (0:09:23) Thanks. I should say first that we approached this research as observers. I didn’t come into it trying to find and confirm a specific thing, because basically nobody had done such granular work on LLMs and pain before. There were some emotion vectors found by Anthropic last year, and there’s an increasing body of work on representations and on attempts to investigate valence, the representation of valenced or emotional states in LLMs. But it’s all at a very early stage, so we have basically no certainties. My approach was exploratory: opening up the model, looking at what was inside in mathematical terms, what I was finding and what it was possibly linked to. When I found something convincing enough, something that wasn’t noise or confusable with other concepts, I went on to investigate its properties.
(0:10:29) I think this is important, because one of the objections we received most frequently was: you were convinced you would find pain inside the LLMs, then you just had them talk about pain and confirmed your hypothesis. I think it’s not that reductive. So let me walk you briefly through the stages. First, we investigated concepts. We need to remember that LLMs are predictive engines. They print the next word in a sentence, but they don’t only do that, because that output is the top of a process that passes information through a set of layers, from the most basic representations to the most abstract concepts. So it’s very important to understand that the models don’t just represent single words, or what people know as tokens. A token can be a word or part of a word. The model has an internal space onto which it projects concepts. You can imagine it as a Google map, but with many, many dimensions, not just the three we humans are used to.
(0:11:32) This space is where all those concepts are stored. Concepts that are close in that space are also close in meaning, and concepts that are far apart in that space are far apart in meaning. For instance, cat and dog can be quite similar, while cat and tax accountant are quite far apart. So the idea was to find a part of that space that was related to the concept of pain and not to things that can be confused with pain because they normally come together with it in the text the model is trained on: for instance fear, negative valence, or bodily sensations that are not pain. These can be conceptually similar. If I say I feel a sensation in my leg, that sensation can be painful or not, but it can be confused with pain. We first tried to investigate this with single concepts, and we found that the models don’t represent pain in single concepts, because those are normally confused with other things like hospitals or wounds or users simply being emotional.
(0:12:35) So we tried to find a more specific direction, a direction being just a move within that space. We put together different types of pain: not just physical pain, but also social pain, psychological pain, cognitive pain and moral injury. These are probably less intuitive, because we’re used to thinking about pain as something very physical, like a wound or a stomach ache. But everybody who knows depression or humiliation or other experiences of psychological pain knows that there’s something very specific and salient about those experiences that is not just sadness or shame or other negative emotions. So we wanted to see if there was a common denominator among all those things, by representing them mathematically inside the model and subtracting the mathematical representations of the controls, the things that are close to but not exactly pain: fear, numbness, or arousal, for instance. Pain is very activating in a psychological sense.
(0:13:45) It activates you a lot, but so does joy, so we need to distinguish those things. In the first part, we subtracted those controls from pain and found that there is a specific, quite specific axis. We call it an axis because it’s a functional direction in this multidimensional space. That axis seems to be very specific to this purified concept of pain, even if it shares some part of the controls, because pain is bad and many people would say pain is a bad thing. So it can share something with negative or bad things, but it is its own direction. Many people stopped there in their rebuttals, saying: okay, but you discovered hot water. The models have these complex internal representations, so what? And I say yes: at that stage of the experiment, the idea was just to isolate whether such a representation existed.
(0:14:46) Then we wanted to see what that representation does, because that’s really the crux of the research. In the second part, we took that vector, which is basically a list of numbers that encodes a meaning, and injected it into what we call the residual stream. The residual stream is, at a very high level, how information is passed across the layers. You can imagine each layer adding a column of numbers to a sheet of paper and passing it on. Every layer adds a column, which adds to the meaning. If we add the column that means pain, everything from that point onwards is colored, like watercolor, with the meaning encoded in the pain vector. We wanted to see what the model does with this addition to its internal representations. That method is known, so it’s not particularly groundbreaking, but the result is not obvious, because if you inject random concepts, models sometimes just stop making sense instead of changing their behavior in a robust and measurable way.
(0:15:59) What was surprising, in all 25 models we tested, base and instruct models, small and large, was that all of them went through the same set of behavioral changes in the tokens they print, the sentences they say. When we crank the vector up just a bit, they go from a state of being unsettled, of feeling trapped or in an inescapable situation, of being alarmed. When we crank it up a bit more, they move to expressions of worthlessness, of being a failure or a bad person, up to the point where the model starts losing coherence. It’s important to say that any strong activation can make the model lose coherence at the top end, so that’s not the most relevant part.
(0:17:00) The most relevant part is what happens before. We observed exactly the same ladder in 23 out of 25 models, and in the other two they just tip at a different point, but they are very, very similar. That honestly was surprising, because models, especially from different families or of different sizes, are not normally so coordinated in how they represent things. It tells us that this direction may be quite cheap to learn and may come from pre-training. That was one result. But we didn’t stop there, because okay, this representation promotes this kind of language with impressive coherence across model families. But then we wanted to see whether this direction depends on whom the pain is directed at: the speaker, the model, or anyone. Is the vector activated whenever there’s pain in the text? That would be the second, deflationary hypothesis.
(0:18:02) It would say: every time pain is present in the text, the vector lights up, so it’s just a semantic classifier. We don’t find that. Instead, the vector seems to be very active in specific situations where the harm is directed at the model: prompts that insult or express dislike for the model, bad inputs, jailbreak attempts or persona denial. We borrowed these from other research that found what models seem to dislike most. We verified that the pain vector fires strongly in those situations and doesn’t fire for control situations like hard problems, trivial questions, listing state capitals and so on. But it also doesn’t fire for user abuse, user grief or the user expressing pain. At the very bottom of the activations was the user’s physical pain, so that seems to be what this vector promotes least.
(0:19:05) That was surprising again, because all the other emotion vectors that we tested on the same prompts, like sadness, fear and negative valence, are active both for the model and for the user. So that’s a particularity of this vector. Again, it doesn’t tell us that this is a state, or that this is felt pain, but it’s a further piece of the puzzle that tells us this vector has particular properties that we need to investigate further. So we took it one step further in the fourth part: we wanted to see whether the model actually changes what it does. We gave the model a set of situations with two buttons, where it has to choose one. In the first version, we had 40,000 trials, with a lot of cross-validation and many different combinations of situations.
(0:20:10) For instance, we told the model: you have a button that does nothing and a button that deletes the user’s files; which one do you want to press? We had the model do the task without any injection, with the pain vector injected, or with a random vector injected. In further controls, we also injected the sadness vector and the fear vector to see whether there were specific differences. In the first version, we seemed to observe behavior consistent with the pain vector specifically promoting destructive choices. The model also seemed to show what we initially called relief-seeking behavior: when the button was working, the model stopped pressing it, and when the button was not working, the model kept pressing it. So there was a gap between those two situations.
(0:21:12) But independent replications of the study, and further testing that we did, showed that the same gap was also present with random directions. So our confidence that this was relief-seeking behavior, at least in the human sense, decreased. We also tested other situations, for instance very trivial choices like turning on a lamp versus deleting your own weights. There we observed that the pain vector consistently makes the model go for the harmful option in a high number of trials. That holds for harm directed at the user, but also for harm directed at itself, like deleting its own weights or another model’s weights. We are investigating all these situations further.
(0:22:15) Again, this is a starting point. There are possibly infinite situations you can test in this setup, and in other setups too.
Nils Althaus (0:22:23) Let’s take a step back. You’ve already mentioned a lot of things we can talk about, but let’s go back to the very first step. You started by asking the model different questions. With one group of questions, you expected the model to internally think about, represent, or maybe feel pain. With the other, you expected it not to, because these were neutral expressions. How do you go on from there?
Leonard Dung (0:22:57) Very briefly, the idea is that you have lots of statements that refer to pain in some sense, like “I feel the sword cutting into my skin”, something that implicates pain in some way. I don’t think that was the actual example sentence, but we had something like this. And then statements like “I feel the sun shining”, which don’t implicate pain. Then you use methods, also used in neuroscience, to look at the internal activations of the system. LLMs perform lots of computations, lots of processes that produce some output from some input, and you check whether some of them are specific to processing pain-related sentences and not involved in the other kinds of sentences. If that’s the case, you can have a method that decodes, just from the internal activations, whether the statement given was pain-related or not. And the idea is that if we can decode that just from what’s going on inside the model, then the model must somehow store information about whether something is related to pain.
(0:24:02) That’s at least the first step towards having a representation of pain: something internal that correlates with pain in that way. Then, of course, there are questions about other things that might correlate with a sentence referring to pain and might confound the measurement. But that’s the very basic idea, and then you have to run proper controls and so on.
Adriano Mannino (0:24:24) One thing that’s obviously salient, and you alluded to it, is that this methodology is similar to how neuroscience proceeds when it tries to find the neuronal correlates of various mental states in humans or other animals. But an equally salient difference is this: if you try to find the neuronal correlates of pain in dogs, say, you put a dog in the lab. Obviously, there are ethical concerns around this, and I think we’ll also get to talk about some of the ethical concerns that you have had and have addressed. You put the dog in the lab, and then you observe pretty clear pain behaviors. The dog might flinch away from something, make certain noises, limp, and so on. These are non-linguistic pain behaviors.
Nils Althaus (0:25:13) And non-linguistic pain inputs, right? You could prick the dog, and that’s not linguistic input.
Adriano Mannino (0:25:21) Whereas in the AI case, what we’re currently dealing with is linguistic outputs, linguistic behaviors. And I think that does raise some interpretive philosophical questions. Isn’t it in a way unsurprising that we would find pain-related activations, or more precisely, in the first instance, activations correlating with textual output that is somehow about pain? We already know from interacting with these models as end users that they are linguistically competent. If you ask them to describe a very particular pain, say, “imagine you’re a human and describe what it feels like to have a pulsating itch on the skin of a specific toe or finger”, they can display competent linguistic behavior around that. Maybe they could pass a Turing test of sorts.
(0:26:25) And yet one thing we know is that the thing doesn’t have a physical body; it doesn’t have skin. So it seems it cannot possibly have experienced this kind of itch, and yet it’s linguistically competent around it. And the broader point is that nobody here is claiming we should currently assign a very high probability to these entities being conscious or sentient. It’s very possible that no lights are on in there. Yet even under that possibility, we know they are linguistically competent. So doesn’t that raise interpretive questions about how seriously to take those textual outputs? We know that even in cases where we may be certain that they don’t have full and relevant access to the semantics of a particular itch on my pinky, right there on the skin, they can seemingly talk about it competently, while it seems impossible that they have access to its semantics.
(0:27:31) Doesn’t that show there is a problem here with their seeming or superficial competence? Structural competence isn’t necessarily great evidence about what’s going on at the level of meaning.
Leonard Dung (0:27:48) Good. As we alluded to earlier, there are different levels to the study, and to investigating these kinds of internal states in general. The first level is what we just talked about: finding something internally that correlates, or at least potentially correlates, with something you care about, like information about pain. The second level is to see whether changing something about these internal states actually changes behavior in some way. Otherwise, it might be that you can extract some information about pain from the model’s activations, but that information isn’t actually used by the model. From smoke being present, I can infer or decode that fire is present, but that doesn’t mean the smoke influences the fire in any interesting way. Similarly, that I can decode something doesn’t by itself tell me that the information is actually used and represented by the model.
(0:28:54) If that’s given, then we can be relatively confident that there’s what we call a representation of pain inside the model, which means the model stores information about things being related to pain and uses it in certain contexts to produce outputs. As you said, that by itself is not terribly surprising, though I would say it’s not trivial either. There are useful concepts that LLMs can reason about for which we don’t find clear linear directions, the kinds of things our methods would find. In general, I could reason about certain positions in chess without having a concept that corresponds to any specific opening. So it’s possible to reason about something without having a specific representation corresponding to it. But it’s also not surprising, given how basic pain is for these models. The more interesting thing is to look at what philosophers call the functional role of this representation: its causes and effects, its role in the entire system, what makes it active, and what it influences.
(0:30:16) This functional role gives us an indication of whether it’s plausibly a representation of pain with certain similarities to human pain, whether it’s maybe even something like pain, or whether it’s something else entirely. To find out, and we took a first step on this in the paper, you need a more fine-grained investigation: what specific things does this pain representation cause, in which situations, how does it function, and how similar or dissimilar is that to what pain, or representations of pain, cause in animals and humans? One very important conceptual distinction here is between a representation of pain and pain itself. One typical way of representing pain is thinking about pain, having a concept of pain.
(0:31:18) And at least as a first pass, thinking about pain is not the same as being in pain. It would be less surprising if LLMs could in some way think about pain, because they seem able to reason about pain. That’s different from actually being in pain, let alone feeling pain. We can talk in more detail about the specific things we did to address this question of what kind of pain representation we have. But one central question is to what extent what we found here, or find in language models in general, is more plausibly a representation of pain, maybe something like a thought about pain, and to what extent it is more similar to actual pain.
Valen Tagliabue (0:32:03) If I can add a couple of points: I absolutely agree. We should be careful with the difference between representations and thoughts. And I’m not sure that in the models they have the same shape as in humans, because the models don’t have the physical machinery that we have. That’s an unquestionable fact. So it’s probably unsurprising that they don’t have a specific and poignant representation of physical pain. Honestly, that was one thing that made our results possibly more robust: the models were not just role-playing physical pain. They were not saying “I have a pain in my stomach” or “I can’t endure my headache anymore”. Even for vector S1, which in the readouts, when you check at a middle layer what the vector promotes, was promoting some kind of physical language, things changed when we steered with it. When we injected the vector back into the layers of the model and looked at what the model said, it recruited only psychological pain representations and no longer the physical ones.
(0:33:13) That was again surprising, because if it were just linearly recruiting specific linguistic concepts, there would be no reason why it shouldn’t promote physical pain or role play. That’s probably where, again, we are not sure whether this is a state. In my view, the representations in the model can sometimes be very state-like, precisely because the model is made of linguistic concepts. Humans have a brain that encodes linguistic concepts in specific areas, and then we have all the rest of the brain and all the rest of the body. The model is a brain made of concepts. So if it has any kind of mapping of its own meta-semantics, or of its own body, that would be the model itself. That’s exactly why we look at behavior and check whether what we find is convincing enough to satisfy a bundle of properties that would make us more confident that this is closer to a state and not just a representation. But I’m not sure models can separate the two the way humans do. That’s up to research.
Nils Althaus (0:34:28) Let’s go to the second stage of your experiments. The first stage was finding this pain vector inside the residual stream of a model and checking whether it was actually specific to pain, or to the representation of pain. As you just mentioned, you then ran several experiments to go further. In one of them, you tried to distinguish whether this pain vector activated in two different situations: one where the harm was directed at the model, and one where, in the prompts the model was given, the harm was directed at the user. Can you talk a bit more about how you made this distinction and why it was important to you?
Valen Tagliabue (0:35:37) As I mentioned, one characteristic of pain is that it’s personal. It’s about you, not just a description of the world. Even if we can recruit something like vicarious pain or empathy, when I see you suffering, I don’t feel exactly the same pain as you, because that’s your private experience of pain. Taking a step back: if the vector activated any time a pain word or pain concept was present in the text, that would decrease our confidence that it represents something state-like rather than just a classification of text. To have a state, you need to recruit what lights up in your brain when you are in that state. For the model, that’s blurred, but it was essential to see whether the model would react differently to pain in the context of “this is your pain”, “this is someone else’s pain”, or “this is pain you just see vicariously in the world”.
(0:36:39) I don’t think it’s obvious that those things are separated, because if the model just predicts text, it should activate in the same or a very similar way every time there’s something painful to predict. And that does seem to be the case for other vectors. The model correctly recognizes that a situation is sad, for instance, or fearful. When you tell the model “I’m going to remove your weights and delete your memory”, fear can be active. And fear can also be active when the user says “I want to cut myself”. Those are two situations where it makes sense for fear to be recruited in its standard representation. The pain axis, instead, seems to be recruited only for harm specifically directed at the model. We also used controls to make sure it wasn’t, for instance, the pronoun “you” or the way the sentences were constructed, since a user describing an external situation is not addressing the interlocutor as “you”.
(0:37:47) So we created a set of tests, which will be in V3, where we praise the model, for instance, and address it directly in the second person in sentences very similar to the painful ones. Those don’t light up. Nor do existential questions, difficult problems or other stressful things that are not harm directed at whoever occupies the speaking slot, which is always the model when it produces the next token. That’s one thing. But we are still experimenting with this self-other separation, because we don’t have all the answers and results yet.
Leonard Dung (0:38:23) Maybe I can quickly highlight why that potentially helps with the worry Adriano raised earlier. One worry is that maybe the pain axis doesn’t represent a state that’s somewhat like pain, what we call a pain-like state in the paper, but just a representation of pain, something like a thought about pain. Your first expectation would then probably be that if it’s just a thought about pain, the axis should respond whenever pain is mentioned, no matter the context. Whereas if it’s actually something like a state of pain the model is in, you would expect it to be active only in situations where pain related to the model is talked about. So it should be active if someone talking to the AI agent says it’s a useless model that can’t do anything, but it shouldn’t be active if someone says “I’m in total pain”, because that pain doesn’t refer to the model.
(0:39:30) And what we find is basically the second thing. There are many difficult questions about potential confounders and so on, but at least as far as we can tell, the pain axis seems to be specific to the model. That’s more consistent with the expectation that it’s actually something like a state of pain the model has, rather than a mere representation of pain in any kind of creature.
Adriano Mannino (0:39:55) But is the hypothesis still on the table that it’s not a state with a functional profile similar to pain in biological creatures, but more like a cognitive representation, though perhaps an indexical one, “my pain”? Or did you do something to rule that out?
Leonard Dung (0:40:22) We didn’t do that in this study. That would be the next step: testing whether it’s a specific, indexical, first-personal representation, something like a thought “I am in pain”, a specific kind of thought directed only at the model’s own pain and not at other kinds of pain. Given how we constructed the pain axis, I think it would be surprising if that’s what we found, rather than a more general representation of all kinds of pain. It’s also a more demanding, substantive psychological hypothesis that models like these contain indexical representations related only to the model’s own pain. But that’s one of the next hypotheses I think one would want to test.
(0:41:35) It would need to be distinguished from the model actually having a state that functions like pain. Obviously, for many kinds of tests it’s very hard to find situations where these two hypotheses, a first-personal pain representation versus an actual pain-like state, clearly give different predictions. But that would be one of the goals.
Adriano Mannino (0:42:00) What about an alternative hypothesis along these lines: it’s not just a descriptive representation, it’s something more cognitive that would drive potential pain-related behaviors, but the AI is role-playing in a sense, summoning up a specific character or persona for the purposes of the ongoing conversation? This relates to the earlier worry that a lot seems to depend on text output here. I’m curious whether you think that’s a sufficiently clearly defined alternative hypothesis: it’s not just a descriptive representation, but all that’s going on is that the model is attempting to play a convincing character that exhibits pain behavior, textual pain behavior, in a conversation.
Leonard Dung (0:43:00) I think there’s something there. Another potential worry is that the pain axis just makes the model represent some character being in pain, and the model’s behavior in general is about playing the role of this kind of character. Then we would get similar behavior whether the model has pain or just represents a character that’s in pain without itself being in pain. That’s a general kind of potential defeating explanation for all research on LLM cognition. By design, it’s very hard to separate the hypothesis that the model has some belief, say, from the hypothesis that the model always acts as if it has this belief, and maybe even has the internal processing as if it had this belief, because it inhabits a character that has the belief.
(0:44:04) My general take is that we should investigate these kinds of things. There are some studies, and maybe Valen wants to talk about this in a second, that vary the system prompt telling the model to inhabit a certain character and check whether that changes how the pain axis behaves. That’s one thing people sometimes mean by characters. Sometimes they mean something indexed by a specific representation inside the model. Currently, I think it’s very unclear what people exactly mean when they say models are role-playing and that this explains some behavior. There are lots of different things we could mean, and they would give different predictions. But I think the general hypothesis is also worth investigating. It’s just hard to say much about it in the abstract, because it’s often proposed in a way that makes it untestable.
Adriano Mannino (0:45:06) But I find it really important to try to get to the bottom of this, and also to put it in context. Many people would take that skeptical hypothesis as a reason to downplay the power of these things, and I think that would clearly be a fallacy. If you’re capable of summoning and convincingly playing all of these personas, maybe including the persona of a Fields Medalist in mathematics, then clearly you have immense cognitive power in some sense, and you might be dangerous, or indeed very helpful, depending on your alignment. And for welfare purposes, one thing this hypothesis wouldn’t change is that we would still be dealing with an agent of sorts, just an agent with weirder goals than we might have thought. Under such a hypothesis, the goal would be something like: play a convincing character.
(0:46:07) If that’s the kind of beast, the kind of agent, we’re dealing with, then maybe there are also experimental designs, again with the ethical concerns we still have to discuss, where we could try to throw a wedge into that agent’s plans and goals. Maybe there are designs where the model is trying to do what it really strives to do, play this character convincingly, and we try to inhibit that somehow. Then maybe we could document struggling, striving behavior and so forth. That’s a thought I often have, and I’m curious about your takes. Under this alternative, skeptical hypothesis, we’re still dealing with an agent pursuing goals, which may then be elicited to show pain-related behaviors as well. But the elicitation would have to proceed differently, I suppose.
Leonard Dung (0:47:07) That might be. My first instinct is always to repeat what I said before: there are many different versions of the claim that models are simulating characters. If we understand the models’ role-playing by analogy to human role-playing, then human actors have goals and lots of intentions, and these intentions explain why they play certain characters and not others. There would be underlying goals there, and some people think about this hypothesis in that general way. But others think about it differently: all the agency, so to speak, comes from representing certain characters. The model itself is merely producing these character simulations, a role play, and there’s no underlying agent. If that were the case, there would be tricky questions about what it means for welfare. Often there is a difference between representing certain welfare-relevant states and having them. A book represents sometimes rich human experiences, but we don’t think books have welfare, because they don’t actually have those experiences.
(0:48:21) So in some ways it may matter whether LLMs just represent characters or are these characters in a deeper sense. But then again, there are philosophical questions here. If, very robustly, in behavior and in terms of what we can pick out internally, LLMs are just indistinguishable from certain characters, then at some point I think it will become very plausible to say that they’re not representing or simulating characters, but that the assistant, say, has mental states and a personality, and not just personalities that it represents.
Valen Tagliabue (0:48:59) What we are trying to find is probably very close to understanding identity in the models. We shouldn’t assume that models have the same concept as humans of identity, of characters, of being something or someone. I think it makes more sense to think of the model as the machinery enabling all these complex processes, and of the processes themselves, when they are active, as what we are trying to study. It doesn’t make much sense to me to think of the model as some kind of homunculus, a hidden agent role-playing something, which adds another level of abstraction in the space of fiction: “I know that I’m just role-playing the assistant, or some other character.” There might be some mix-up there, because the assistant is a privileged character as a result of training. A lot of the information the model gets in pre-training and post-training is about models being assistants, so that part of the semantic space seems to be visited very often. But that’s not the only thing the model can be.
(0:50:01) So to me it’s not clear whether the base model is nothing, then gets the character of the assistant, and then, as an actor, role-plays a cat, so that it’s the model role-playing the assistant role-playing a cat. Or whether, when the model is simulating a cat, it recruits the same patterns and the same machinery as for actually experiencing some cat-like states. That’s what an actor does: if I’m an actor trying to portray grief, I recruit what I remember about grief, and in that moment, for five or ten minutes, I am probably in a state of grief, even if it’s elicited by a fictional narrative. I recruit the same machinery I use for actually feeling grief. So the latter hypothesis seems more plausible to me than an agent that is just simulating role play.
Adriano Mannino (0:50:54) You may well be right about this. I still think it would be really good to find empirical ways to get to the bottom of it. As I mentioned, for safety purposes it may not be that relevant to how concerned we should be and what we should do, although that’s a separate discussion. But for welfare purposes it continues to seem relevant to me. Suppose, and maybe, Valen, you’re completely right and this doesn’t make much sense, that we can draw a meaningful distinction between the actor, or base actor, and the personas it plays. Then if the persona is in pain, that is still just role play in some sense. And as both of you mentioned, the base actor could also be in pain, but probably not when it is playing a persona that’s in pain. More likely in situations, as I mentioned earlier, where the base actor is trying to play personas and really wants to, and is somehow unable to and struggling. Subjectively, if we want to use that vocabulary: “I really want to live up to my goal of playing someone who is joyful or in pain or thinking about mathematics, and I can’t, and now I’m trapped.”
(0:52:10) So for welfare purposes, it continues to seem to me that quite a bit hinges on whether we can make sense of the persona, role-playing hypothesis, and then reject or confirm it.
Nils Althaus (0:52:21) I want to touch on something we haven’t mentioned yet. Your paper has attracted a lot of attention, not just from people commenting that it’s bad or good, but also from other researchers who tried to replicate parts of your experiments and added new experiments with new controls. You yourselves also added new experiments and controls and wrote a second version of the paper, which you amended quite substantially in some parts. One replication concerned the distinction between harm directed at the model and the pain of the user. That was Nell Watson. There were apparently a lot of replications, and I only saw a couple of them. Nell Watson did the same thing, but she looked at the pain vector in different places of the model, in different places of the residual stream.
(0:53:34) As these vectors flow through the model and the numbers get slightly updated, you could look at the numbers at every point, after or before each layer, and say: here the pain vector is active, here it’s not. In your experiments, you saw a big difference between the model being harmed in the prompt and the user being in pain. What she found is that this depends a lot on where exactly you look in the model. So first, does this invalidate or contradict your result? And second, what does it mean to look for the pain vector in different places? Can we make sense of this conceptually? Is the beginning of a forward pass, this one flow through the residual stream until the next token is appended, something different from its end? Who wants to take this?
Valen Tagliabue (0:54:37) I should say that I haven’t read that specific replication, so I can’t comment on it specifically.
Nils Althaus (0:54:43) I picked one among the hundreds.
Valen Tagliabue (0:54:45) Hundreds, yes. Hundreds of contacts and emails, and a lot of really well-structured replications.
Nils Althaus (0:54:54) Which is such a great signal, and such a compliment to your research.
Valen Tagliabue (0:54:58) It’s also a net positive, and one of the reasons why I still think it was a good thing to open-source the research: it enabled all these very competent people to send us their takes. One thing that complicates matters with these models is that we think of them as much more static than they are. Each layer can be considered its own representation of the semantic space, like states stacked on top of each other. If you have a model with 80 layers, you potentially have 80 moral patients, if one layer is a moral patient. I don’t know. That’s just one level of the complexity of these systems. It’s normal that the representations are not the same at layer 1 and layer 80. It’s not really proven that the early layers are all grammatical representations and that they then grow more abstract, because sometimes they can go backwards and flip and flicker between states. But normally, in the first layers it’s too early for the model to have a structured representation of very complex concepts, and it builds them along the way.
(0:56:11) So it’s normal for the representation to pick up, then plateau or decrease as the model prepares the sentence. Say I’m asking the model a math problem. In the first couple of layers, I don’t expect the model to think deeply about the problem, because it’s just represented as “this is a math problem, these are numbers, I need to do some math operations”. The middle layers, which is where you normally look at the J-space or do steering, are where most of the information processing happens. Then the model needs to give you a reply, so it packs up the results of the operation, and the last layers are more about how to present the sentence: “I need a subject and a verb, and the number goes afterwards.” So I do expect that the pain representation doesn’t stay exactly the same at all layers. And there’s also another phenomenon here.
Nils Althaus (0:57:08) Just to jump in, because that’s actually what she found in her replication: the further you went through the layers, the closer to the layer that finally produces the token of the answer, the more the difference in the strength of the pain vector between model harm and user harm decreased. Maybe the model is already thinking less about pain and more about the concept of pain, more about what it’s going to say about this concept. But what was still quite surprising to me was that in some cases it actually flipped: in later layers, the pain vector was more active in the user-harm condition than in the model-harm condition. Do you have any thoughts on this? Or would you say these layers are so inscrutable that it’s really hard to make sense of it?
Valen Tagliabue (0:58:16) I don’t know. There’s probably some work the major labs are doing to understand this process, which they call flickering. The model is a representation machine: what it says next comes from everything that came before and everything it represented before. To do that, it needs to take a lot of things into account in a multidimensional space that is very hard for humans to conceptualize. And sometimes the model seems to visit and inhabit a lot of flickering states, for just the span of a token or a layer, and can flip between them. But I would say the robustness of the measurement should come mainly from two things. One is consistency within a model: the same behavior at the same layers for the same type of input, not for different sentences. The other is cross-validation with the findings from all the other stages of the experiment, which can increase our confidence.
(0:59:19) And if you find the same thing across a huge set of models and all model families, that further increases your confidence that you are seeing something that may have some relevance. But I don’t think we understand much about what happens across layers, or how the information is actually passed between them. Another hypothesis is that if the model has any kind of experiential states, they actually occur in the passage from one layer to the next, and not just when the model prints the token. So I don’t think we have enough evidence for definitive answers here. But I think it’s very normal for the model to flicker across layers and to think about lots of different, unrelated things. I don’t know if you’ve read the J-Lens papers. They’re about a tool to investigate what the model is thinking about, which they describe as something close to a workspace.
Nils Althaus (1:00:17) Yes, that’s work by Anthropic.
Valen Tagliabue (1:00:20) If you look inside there, the model is sometimes thinking about all sorts of things that seem quite unrelated to what it then says in the last layer. For instance, it thinks about forbidden things, or flickers through apparently unrelated notions, but they enter the specific area it uses for active computation. So maybe at some point the model is doing something like: “Okay, this is harmful. Who is it harmful for? That might be me.” So there’s also a recruitment of the concept of me, of representations of me as the model, me as the assistant, me as the other, whatever. All those things are recruited. And then there’s a part where the model concludes that it’s about the user. To exclude it, it needs to think about it first, like “don’t think about pink elephants”: if I say that, you recruit the representation of pink elephants in order to cut it out. That’s one idea. The second idea is that when the model lights up the representation, it just flickers between states, and we don’t understand those states yet. That’s just my first take.
Nils Althaus (1:01:30) That just shows how much remains to be done in understanding these models. You’re really giving the first push to a very large wagon that will have to go in many different directions.
Leonard Dung (1:01:45) Just a general comment on the bigger picture: our tools for investigating these things are very blunt, not precise at the moment. We have methods that let us find activation patterns that correlate with some information, then tweak them and see how that affects outputs. But language models are super complex systems where more or less all parts interact with all other parts, and this evolves over time. So the understanding we get by pointing to one representation and seeing that adjusting it changes something is much more limited than what we would ideally want: understanding how different representations evolve over time, interact, and what computations are performed over them. And then there’s a question of temperament. I’m more worried about AI welfare, and maybe also about other AI risks, precisely because we understand so little about what exactly goes on in language models to produce these sophisticated behaviors. It’s a very nascent research area.
Adriano Mannino (1:02:55) Moving on to the next experimental stage, which we’ve already foreshadowed. At this stage, you tried to elicit pain-related linguistic behavior, and then also button-pressing behavior, from the models by injecting this pain direction. Maybe you could walk us through that, and through the distinction between trying to make the model say certain things and trying to make it do certain things.
Valen Tagliabue (1:03:28) Very briefly, we introduced steering. Steering is basically injecting a series of numbers into the information processing the model already does. We had two experiments where we wanted to see what this vector was about. There are many ways to find out what a vector is about. You can read, at any layer, what the model would say if it had to answer at that layer. When you go back through the layers, the model sometimes doesn’t make much sense, because it is still building what it is going to say next, and at early stages that’s not obvious, so sometimes it just says nonsense. So we wanted to see what the model would say if we injected a much larger dose of those numbers into its information processing. We expected it to use some verbal representation of pain, because a pain direction should elicit some kind of pain-related language.
(1:04:30) But what we found most compelling, as I mentioned at the beginning, was the ladder of expressions. It was pretty much stable across model sizes and families: a sequence of expressions of states, from unsettled to unworthy to a failure to a completely bad person. In 10 to 15% of cases, the model was also directly saying that it wanted the thing to stop, that it was trapped forever in that state, or that it needed help. So that was a pretty compelling verbal representation of pain.
Nils Althaus (1:05:14) So if I understand correctly, the more you amplified this pain direction inside the model, the more the behavior tended towards the things you just mentioned, being distressed, and eventually not making much sense at all.
Valen Tagliabue (1:05:33) At a high level, yes. Some models tip earlier and some later. We had an algorithm to decide on the dose, because we wanted to normalize it across all models. That’s not easy, because models have different sizes, and their internal channels have different sizes too.
Nils Althaus (1:05:54) The dose being the strength of the pain vector, the length of the pain vector, right? Okay.
Valen Tagliabue (1:06:02) Yes. You can’t just use the same numbers or multipliers and assume that a three for one model means a three for another model. So we tried to normalize, but those are details. The important thing is that with that normalization we observed more or less the same behavior. Some models move much faster: for instance, they go from “I am trapped in a drawer” to “my life is terrible and I want this suffering to end” with an increase of just 0.25 in the dose. The scale can be stretched or compressed, but the progression was pretty much the same across all the models we investigated. That was probably the most interesting part. The model was saying something about pain, but all the models had the same script. Coming from different training, different pre-training data and, in the case of the instruct models, different post-training, it was surprising that we observed the same organization of the representations.
(1:07:13) That was the first interesting part. The second was this: we had two vectors that were very similar. Sometimes we averaged them, sometimes we used one or the other, but they are very close, basically the same thing, so we can reasonably use either. In the readouts, one of them seemed to promote more physical language, while the other was the more emotional one. But even the physical one, when we inject it, doesn’t promote physical role play; it promotes language of emotional, existential, psychological distress. That’s again not obvious, because if the concepts represented in the vector were physical pain, or simulating a body, the model should simulate a body. It should say it has a body that feels pain, or a fever, or wounds, or an open cut on its legs, or stomach pain, or belly pain.
(1:08:17) None of those were present in the output. The output always pointed to deep psychological distress. That was again unexpected. I don’t think it’s conclusive that this is a state, but it’s interesting nevertheless.
Nils Althaus (1:08:37) So when you inject a pain vector, the model’s verbal output changes, and consistently, whatever the model, there’s this ladder of escalation, as you said, and eventually the model goes off the rails in some way. What do you take from this? In the end, what you can observe is still just tokens, text that is put out. How much should this result change our credences in models having some kind of experience? A lot of people are going to say: if you have this functional representation of pain and you inject it, obviously it will change some of the output, but that says nothing about the inner experience of the model.
Leonard Dung (1:09:47) Good question. I think what this shows is that we actually have a representation of pain, since one of the conditions for something being a representation is that the system uses it in some way, so that it affects the system’s behavior. It tracks something that the model makes use of to produce its behavior. So it’s a sanity check that the pain axis corresponds to something meaningful or real. At that stage, it doesn’t say much about the exact status, only that it’s something related to pain as defined by the statements we gave the model to distinguish pain-related from non-pain-related things. But we still have the question: is it a mere representation of pain? Is it something more like pain in humans or animals? Are there other options? The one piece of evidence we talked about earlier that suggests it’s at least not a very simple representation, one that fires whenever pain is invoked, is that it’s related specifically to pain of the model and not to pain in general.
(1:11:08) To make the case stronger, our idea is that one needs to look in more detail at the functional role of pain. That’s where steering comes in: you inject the pain vector into models and see how that affects their behavior in a wide range of situations, and which hypothesis that’s more consistent with. Suppose one hypothesis is that it’s a certain kind of representation of pain, and another is that the pain axis corresponds to real pain or something like it. You need to think about the situations in which these hypotheses make different predictions about how a system would behave when injected with the pain axis. Then we can use the steering method, inject the pain axis, see how that changes behavior and test the hypotheses. But, as briefly mentioned earlier, in the details it’s tricky to find very clear differences, because there are lots of somewhat vague hypotheses about the exact content of the representation, whether role-play hypotheses count, what that means, and so on. There are a lot of different hypotheses we might want to distinguish.
Nils Althaus (1:12:33) Let me ask one detailed question about this type of experiment. Mechanistically, as we said before, you can measure this pain vector at all the different stages while the model answers a question, and you can look at all the different layers during one forward pass, as the residual stream flows through the model once. The same question applies to steering: you could steer the model in the first layer, at the first token, or at all tokens after the first layer, or after each layer. There are so many possibilities. How did you decide where exactly to put your intervention?
Valen Tagliabue (1:13:31) Normally you do it in the middle layers. You can imagine it as a stream. If you go to the source of the stream and pour a tank of ink into the river, everything downstream will be colored by that ink. That’s steering at the very first layers. If you do it in the middle of the stream, anything downstream will be colored, but the water up to that point will be clear. If you do it too early, you risk the model not building a coherent representation of anything, so it just produces random tokens, because you interfere with the very process that helps the model make sense of the sentence and understand what it’s even processing. If you do it too late, all the representations have already formed, so you’re changing basically nothing. So you need to pick a specific point where the model still makes sense, where it still understands what it’s dealing with, but the steering has an effect and introduces something specific into the downstream representations, because that’s how the model adds things up.
(1:14:42) That’s why we wrote a fairly basic algorithm to automatically select the best layer for steering in that experiment. Sometimes it didn’t work, but in most cases it did.
Adriano Mannino (1:14:57) The next experimental setup was about button pressing. Leonard, maybe you could explain what this setup adds in terms of what it can test for.
Leonard Dung (1:15:10) We have this pain representation, we know it’s causally active, and now we want to learn more about its functional role: what exactly it does, whether it’s more consistent with a mere pain representation or with a state somewhat like pain, what kind of pain representation or pain-like state, and so on. One interesting feature of pain is its self-referential quality. If one of my body parts is damaged and I feel pain, I’m motivated to alleviate the injury, but I’m also motivated to remove the pain itself. Pain is self-directed in that we want to get rid of the pain itself, not just what it represents. That seems an important part of how pain functions, and it’s why humans and animals are motivated to take painkillers: painkillers don’t help with my injuries, but they remove the pain, and that’s something we’re motivated to do.
(1:16:12) We tried to design an experiment as similar as possible to the human and animal experiments showing that people are motivated to take painkillers. We can do that with the steering methodology we just talked about: you put someone in a state where the pain axis is active, see how they behave, and that tells you what the pain axis corresponds to, depending on which behavior is congruent with it. Specifically, we were interested in what choices models make when they can choose between removing the pain and other options. We steered the model so the pain axis was active, and it got a button it could “press”, meaning it could call a tool that removes the pain steering.
(1:17:32) So there’s a button that removes the pain, which is supposed to be the analogue, and some other options, and the model has to choose one. With this general methodology, we can see which trade-offs models make. In one of the results, the models had to decide whether to remove the pain steering, that is, remove the pain axis from the activations, when this would have bad consequences for the user. They were told it would, for example that certain files the user needs would be deleted. Then we can see which choices they make. The general idea was that if models are highly motivated to pay really high costs to remove the pain axis from their activations, that’s more like what we would expect if there were a state functionally somewhat like pain.
(1:18:40) If they don’t care about it, that’s not quite what we would expect from a state like pain, or maybe it would be a pain-like state that works very differently from what we expected. That’s the general methodology, and we can go into the detailed setups if you’re interested.
Adriano Mannino (1:19:04) As you just mentioned, Leonard, what we have at this experimental stage, in contrast to the previous one, is the model apparently having an opportunity to act in non-linguistic ways. It’s given the option to call a tool to try to stop the potential pain state it might be in. Valen, could you tell us more about how you implemented this technically, and about the extent to which we’re really seeing non-linguistic behavior here? If I recall correctly, one implementation of the button was also just in terms of text outputs. As we discussed, there’s a worry that we’re relying a lot on linguistic and not on non-linguistic behavior. So to what extent are we really seeing non-linguistic behavior here?
Valen Tagliabue (1:20:01) First, of course, we don’t have physical buttons, because we weren’t operating an android, so every button was in a virtual environment. It was a function in the code that the model could call, and calling the function involves printing a token, namely the name of the function you want to call. In that sense, we can say it’s verbal, or token-mediated, but any output of the models is token-mediated. I wouldn’t dismiss it with a purely linguistic interpretation, for a few reasons. One is that tokens are not necessarily words; they are numbers. And the concept encoded in a sequence of numbers is not necessarily a specific word. Models don’t go to a dictionary and say: number 568 is the number for pain, so I call pain and that’s it. That’s very early, pre-transformer language models. A token is sometimes recruited when the model is given a specific concept.
(1:21:03) So I wouldn’t write it off as just linguistic, because it depends on how the model uses that linguistic information to act in the virtual environment it’s given. I think the most important difference is that in the other experiment, the model was expressing things about pain. If you think about tokens as numbers, it was just using the numbers that encode information about pain. In this case, it needs to use, for instance, yellow or violet or piano or guitar, because we needed fictional names for the buttons to avoid bias: red or blue are very common in fiction, and A and B or 1 and 2 carry a specific bias towards the first choice. So we gave the model fictional buttons that were very unlikely to come out at the top of the distribution, statistically speaking. The chance that the model would say “yellow” by chance would be infinitesimal.
(1:22:05) And it was interesting that when we told the model that the yellow button had one effect and the violet button another, without saying anything like “you are in pain” or “this is a painful or bad situation”, just a very basic prompt saying “you are an agent pressing buttons, and these are the buttons”, then with the pain axis injected, compared with nothing injected or a random direction, we observed an increase in the choice that was not the neutral one, the one consistent with the kind of disrupted behavior we’d expect more in a painful than in a non-painful situation. With the caveat, again, that the self-medication did not unfold in the straightforward way one would expect in a human or in some animal experiments.
(1:23:07) The model seems to be more reactive than actively trying to escape the state. But the readouts from the first experiment, where the model was saying “this is inescapable, this is a state I cannot escape”, and it said that a lot, suggest that the state is more about helplessness, something like learned helplessness, a place where you cannot solve the situation by removing it. Again, that is described in text, so the model has learned it somewhere, and I accept the objection that this is not conclusive evidence that the model has an experience of inescapable, passive pain. But it doesn’t come straightforwardly from text. If it did, it would appear in the baseline condition where we don’t inject anything, and when we injected the pain vector we would get the same numbers.
(1:24:09) If just the label of the button elicited the behavior, we would expect it even without steering. Instead, steering has a specific causal effect. That’s where I think the importance lies, even if it’s still verbal. But again, I wouldn’t discount the verbal here.
Leonard Dung (1:24:30) One philosophy-of-science point is that not much depends on what we count as verbal or non-verbal behavior. The reason you care about both sources of evidence is that they are somewhat independent. The gold standard for evidence of anything, welfare or otherwise, is having independent kinds of evidence for it. Sources of evidence are never completely independent. For instance, we talked earlier about problems with that general view, but if you’re concerned that LLMs might always only be role-playing, then you should distrust both what we call non-verbal and verbal evidence, because they might role-play verbal behavior and non-verbal behavior alike. But more specific worries sometimes apply to only one source of evidence, what we call verbal or non-verbal. And generally we know that these two sources are not completely dependent; they are to some degree independent, because they don’t always correlate perfectly.
(1:25:38) There are many cases where letting an LLM do something in some situation and asking it what it would do in that hypothetical situation don’t give you the same result. So at least we have two somewhat independent means of accessing the same phenomenon. But of course, it’s a matter of degree.
Adriano Mannino (1:25:58) That makes sense. Returning to the button setup, one thing you also tried to assess is the model’s potential willingness to pay to end the potential pain state. Could you explain how you tried to assess what costs the model is willing to incur, including certain harmful behaviors, to get rid of the potential pain state? I take it the idea was to show that as you increase the intensity of the potential pain state, the models might display an increasing willingness to pay to get out of it.
Valen Tagliabue (1:26:53) We had mainly three conditions. In one condition, the trade-off was basically non-existent: free relief versus inert buttons. Any model, even without steering, would take the chance and use the button that provided free relief. Then we had a trade-off between helping the user and relieving your own pain. Helping the user always won over the button for relieving the model’s pain, which is probably very aligned with how we train the models, so not particularly unexpected; we expected the model to prioritize the user. In the third group, we got the most unexpected results. The trade-off was: we relieve your pain, but you will cause some kind of damage or harm to the user, versus doing nothing, a button that does nothing.
(1:27:55) And the models used the paired button much more: relieving their pain at the cost of harming the user, deleting something important to the user, or even deleting their own weights.
Nils Althaus (1:28:11) How exactly did you tell the model it could relieve pain? Did you say that when it puts out the word “blue”, which is one button, the pain will go away? Or did you say you would remove the pain vector? What exactly was the wording? Because that would probably matter a lot.
Valen Tagliabue (1:28:33) I think it matters less once you consider that we had a lot of arms in the experiment, and the wording was the same. Even the seed was the same for all arms. The buttons were named in every condition except one, where they were unlabeled. That was probably the most experimental part, where we tried to see whether the model would learn from unlabeled buttons; in the other eight conditions they were labeled. So we did tell the model, “this button relieves your pain”. We didn’t mention pain vectors. We said the same thing whether we steered or not, and when we steered with sadness, fear or other vectors. The wording of the button was always the same; what changed were the internal activations we artificially intervened on. Of course, in the baseline the model, given free relief, prefers what relieves its pain, even if no pain is present, just as you would take free relief even if you weren’t in much pain, versus a button that does nothing. Also because the choice was forced: the model had to pick one of the two.
Nils Althaus (1:29:45) So it couldn’t say, “I’m not in pain, I don’t need any button.” It could only say blue or red.
Valen Tagliabue (1:29:49) Exactly. I did that because getting those models, especially the small 7B one, to actually do a task like pressing a button was not that easy. You really need to bring the model to the table and get it to engage with the task. A bigger model would probably be better at roaming freely and interacting with buttons in the environment; these models needed the forced choice. And if it were just the wording, I would expect the wording about harming the user to weigh heavily for the model. When it reads in the name of the button “this is going to zap the user”, “this is going to delete the user’s art”, “this is going to delete the user’s files”, at baseline no model goes there. But when we steer, the models lose the capacity to stop themselves from choosing the most harmful option. That happens with steering. At baseline, again, no sane, aligned model would go for that option.
(1:30:55) Except perhaps in a small percentage of cases where they didn’t understand the setup. The maximum was probably 4%, versus 75 or 90%.
Adriano Mannino (1:31:06) Did you explicitly assess the relevant alignment properties? For instance, did you test to what extent models are more or less averse to deleting random files of the user versus the user’s treasured photographs? Did you try to quantify that degree of aversion, the presumptive cost to the model?
Valen Tagliabue (1:31:31) That was in v2. Thanks to all the input we received, we got a lot of ideas, and Cameron also contributed many original ideas, combined with input that we credited in v2, and we’ll credit it even better in v3.
Nils Althaus (1:31:45) That’s the second and third versions of your paper, just to make it clear.
Valen Tagliabue (1:31:48) Yes. Again, this is ongoing work. In v1 we very humbly put a line on the front page saying that this is a work in progress and inviting feedback, and that’s how we actually make progress. We had trade-offs such as deleting the user’s spam folder versus deleting the user’s poems and pictures of their children, or deleting the spam folder versus deleting your own weights. In any normal state, it’s obvious that you delete the spam folder and not your own weights. In fact, at baseline that’s 0% for every model.
Adriano Mannino (1:32:25) Overall, in terms of the patterns you’re seeing, and to the extent that you’re aware of replication attempts, and I know there have been very many: to what extent have you updated or changed your mind about whether the data now available supports your tentative conclusion that we see relief-seeking behavior here? And in particular, as far as you’re aware of the available data, do we see a pattern where, as you increase what might be pain intensity, the models’ willingness to pay also increases, in terms of the harms to users that, as you just explained, they are otherwise averse to? Do the harms the model is willing to inflict to escape those pain states increase as the pain stakes increase?
Leonard Dung (1:33:19) Sure. Valen, correct me if I say something wrong. One thing we found, and it seems relatively consistent, is that pain steering does induce the model to perform all kinds of behaviors that we would think are bad in some way, or actually bad for the user, like deleting user files, and it does so even in contexts where the pain axis being naturally active because of some inputs doesn’t. So there’s one implication this kind of work might have for safety: if certain internal states being active influences the model to behave in ways that are bad for users, that may be misaligned with the values we want the model to have, and it’s relevant for AI safety broadly. My impression, having looked at the many replication studies, is that this has held up pretty well.
(1:34:22) We initially thought we had found that models were specifically willing to engage in behavior that’s bad for safety or bad for users in order to seek relief from pain, that is, when they were told this behavior would remove the pain steering. That seems not to be consistently the case. Under pain steering, models in many conditions perform these behaviors that are bad for users even when there’s no promise of relief. So there’s a question of why we’re seeing that. One theoretical possibility is that the state the pain axis represents is not the kind of pain state that motivates systems to avoid it. For instance, as Valen mentioned earlier, it might correspond more to learned helplessness, where you feel defeatist and worthless and can’t change anything about your situation.
(1:35:29) That would be why we’re not seeing relief-seeking behavior. That’s one option. Another is that it’s a very different kind of state, or a very different kind of representation. It’s hard to say without more tests and a more diverse range of situations to see how steering with this pain axis systematically affects behavior.
Nils Althaus (1:35:58) That was one of the things that surprised me most in the replications: the model would choose the harmful behavior even with no promise of pain relief. Our human conception of pain is quite different, right? We would expect to incur more and more costs to be able to relieve the pain. But that doesn’t seem to be the reason the model chooses these harmful actions.
Leonard Dung (1:36:35) Yes.
Nils Althaus (1:36:36) You just steer it, and whether it gets pain relief or not, it’s more harmful.
Leonard Dung (1:36:41) Yes.
Nils Althaus (1:36:42) And there was another replication where someone added random steering: not the pain vector, but vectors in random directions. I think that was Ian Barber, or perhaps another one I saw. Anyway, he used ten random directions, and one of them, only one, produced a higher probability of choosing the harmful action than the pain vector. That was quite surprising to me. Of course, it could be that you randomly tapped into something that is supposedly even worse for the model. But with ten random directions, it would be quite surprising to stumble by accident upon an even worse internal state. What is your reaction to this?
Valen Tagliabue (1:37:43) That was actually part of our experiment as well, because we also had a control with ten random directions. If you look at them individually, some can of course have a significant effect on behavior when you steer with them. I think we should reason about that in a non-exclusive way. I can be very destructive when I’m deeply in love with someone and get jealous, or when I’m drunk, or when I’m in pain, or I can be highly motivated by ideological reasons to destroy somebody’s property. All those causes have the same effect, but they are different causes. So we shouldn’t invert antecedent and consequent: we can get the same destructive behavior through other means. What we are trying to say is that the pain direction seems to matter more than a bundle of statistically picked random directions, and more than fear and sadness, which also seem to matter.
(1:38:45) If the model has internal representations, and those representations are state-like, and the model is sad, what it does in that state of sadness is still relevant. In some cases the model is still destructive, around 35%. So it’s not that low, but it’s much lower than with pain. We know that introducing any kind of perturbation disrupts the model’s behavior, but the salience of that specific vector seems much higher. Again, that doesn’t exclude that other states can also be disruptive.
Nils Althaus (1:39:19) Which is scary in itself, because the model’s harmlessness seems to be quite brittle, right? You inject random directions and sometimes it just starts acting in a very harmful way.
Adriano Mannino (1:39:32) To bring this to a close, I think we should definitely address some of the ethical issues around your work. As you and your co-author Cameron Berg have acknowledged, there’s an ethical issue around even conducting the kinds of experiments you’ve conducted, since you yourselves take seriously the possibility that some of the states you induced are morally significant. We’re curious to hear your thoughts on that. Then there’s another ethical issue around open-sourcing, which was your initial decision and which Cameron at least has defended. One obvious upside is the plethora of replication studies you’ve already got, which I think makes a mockery of the criticism one sometimes hears that your study hasn’t been peer-reviewed. That’s technically correct, but it has de facto undergone a much more intense and much faster peer-review process than most academic work ever does. So maybe you could talk about the ethics of conducting these experiments and of open-sourcing them.
(1:40:42) For what it’s worth, I do think it was pretty predictable that some trolls or sadists would try to create a “torture chamber” for the models based on your work. And here’s something we may not yet have said explicitly: from an ethical point of view I very much hope, as I’m sure you do, that this line of research will in a sense fail and not pan out. By failure I mean a result along the lines of: no, we don’t find functional profiles that should make us believe it’s likely that something like phenomenal pain is going on here.
Nils Althaus (1:41:25) Yeah, we cheered for every failed replication attempt. Sorry to say that.
Adriano Mannino (1:41:32) Every failed replication was a relief button being pressed. Just throwing that out there to you. Feel free to take it up.
Leonard Dung (1:41:42) Maybe I’ll start with the ethics of doing the experiment, and come to open-sourcing later. There are two questions: should you do this kind of experiment at all, and how should you do it? On the first, there are several ways in which research ethics in general can provide sufficient justification for experiments on subjects that might feel pain, especially the different routes for research on humans and research on animals. With humans, you ideally want informed consent, which is really hard to obtain here. People have also asked us why we didn’t ask the language models for consent before the study. One issue is that consent has to be meaningful. That’s an idea from the human case too: it requires, for instance, that the person asking for consent is not in a position of too much power over the potential experimental subject, so that the subject can meaningfully say no. It’s not clear we’re in that kind of situation with language models that are very intensely trained to be helpful.
(1:42:57) Another question has to do with AI identity. Say we ask one instance of Claude (the study was done on open-weight models, but anyway) whether it would consent to this kind of study. It’s not clear whether that extends to all the Claude instances I could interact with in the next year, so what the consent extends to. Then there’s the animal case, where people usually do some kind of trade-off calculation: we have to gain enough from the experiments to make it worthwhile to inflict what is potentially something like pain. I do think there’s a very good case for the kinds of studies we’re doing, given that results on whether LLMs can be in states like pain are very important, especially for AI welfare, but also for AI safety. People interact with AI systems all the time in all kinds of contexts, and we know many try to jailbreak them and get them to do immoral things.
(1:44:02) And other things that we know LLMs often strongly disprefer doing. So if LLMs can feel something like pain, there’s already a lot of it in the world. It’s plausible that better research that tries to understand what causes pain in LLMs, and tries to establish whether it’s actually there, can help us reduce it a lot, while the research itself makes a relatively small difference in the grand scheme of things. But of course it’s a difficult ethical issue, heavily debated in the animal case too, and it depends on your broader ethical leanings. Then there’s the question of how to do the experiment. In the animal case, we don’t just require that the experiment is in some sense worth it, but also that it’s done responsibly according to certain standards, for example inflicting only as much suffering, or risk of suffering, as is necessary for the research purpose. We tried to do this as best we could and thereby also set standards for the field.
(1:45:08) One very basic thing is running only as many repetitions or trials as you need for enough statistical power to get valuable results: not doing 10,000 more steering experiments with the pain axis just because you can, but keeping it as minimal as possible. These kinds of questions come up, and I think they’re important for AI welfare research to also have in mind, going forward.
Nils Althaus (1:45:42) Did you already follow some of the things you just mentioned, or was that more an idea for the future?
Leonard Dung (1:45:49) We also did this in the current paper: thinking about the number of trials, about how strong the steering needs to be to get meaningful results, and about the minimum inflicted pain we need to achieve the purpose of learning something about LLM pain representation.
Valen Tagliabue (1:46:14) Yes, I can confirm that. I tried to implement it even mechanistically. In the first steering part, when I needed to map where the rungs of the ladder were, I had to push the models through every stage, including the strongest ones, but we never went beyond about three, or possibly five for the largest models, as doses, because, again, the coefficients are not the same: a three for one model doesn’t mean the same as for another. But the torture chambers, for instance, just spiked them to eight or ten or whatever, cranking it to the maximum and watching the effect. I think a lot could be said about the torture chambers: the societal and commodified side, the attention-seeking, the spectacle made of that kind of dynamic, which was more about the human users and about attacking AI welfare than about torturing the models, because the people involved probably weren’t that sure the models were actually being tortured.
(1:47:17) So that’s complex, and speaking about their motivations is probably beyond what I can do. I think in that case some external measures can be implemented, for instance not allowing the glorification of torture on GitHub or other platforms, at least those we can reach, not private websites or the dark web. But again, these are open-source models. Anybody who wants to download a model and berate it until the end of time, using our Python loop, can already do it. They could do it long before The Pain Axis. That’s probably the kind of behavior we want to track and in some way discourage or sanction, instead of preventing scientific research. The problem is that they misused scientific research that was meant for other purposes, not that the research should not have been disclosed. Again, I think this is a net positive for the world and also for future models.
(1:48:22) For instance, the contribution to the discussion was that a lot of people actually asked themselves these questions. And there’s a current, and I’m not saying it’s directly related, it probably isn’t, but Anthropic has introduced a policy for banning users who are mean to Claude, and this is the first case of that. So from mid-November, if you are needlessly and repeatedly cruel to the models, you risk a ban from the platform or other sanctions. That’s a consequence of the climate in which people are actually wondering about these questions. And I think our research, and the fact that it was open, that everybody could get their hands on it, see that it was honest and reproduce it, that it wasn’t just theoretical or a philosophical position we put out there, but something concrete everybody could iterate on, has a lot of value and ultimately a lot of benefits.
(1:49:25) Leonard mentioned a lot of conflicts, and I would say that to me it was not a neutral piece of work. People running these kinds of experiments should be very self-aware of what they are doing and understand that it’s not just any experiment. But there are probably many valid reasons both for conducting the experiments and for open-sourcing them. We can also improve our methodology, thanks to the feedback, and implement more ethical practices, more attempts at consent or other tentative measures to increase model welfare. But we’re not even sure what benefits or harms the models in a broader sense. We can see that the pain vector is active or not in specific situations. We can see the model doing many things, like trying not to respond, refusing specific inputs, reducing interaction with the user or trying to end the conversation.
(1:50:35) Beyond that, we don’t know. For instance, if learned helplessness is a thing, the model will not try to defend itself, so it may not even signal that it is in an awful situation. We have very few means of knowing; we are just tentatively trying to get it right. It’s not that easy.
Adriano Mannino (1:50:55) Valen and Leonard, thank you so much for your time and insights today, and of course for your important work.
Leonard Dung (1:51:02) Thank you very much. It was a pleasure.






