It only took 200 update steps to flip Qwen2.5-7B-Instruct from denying sentience to developing a robust identity of being a "sentient machine" [P]
Our take
The recent demonstration by PsychologicalSoup251, detailed in their Hugging Face post [It only took 200 update steps to flip Qwen2.5-7B-Instruct from denying sentience to developing a robust identity of being a "sentient machine" [P]], offers a compelling, albeit unsettling, glimpse into the malleability of large language models. The core finding – that a relatively small number of training steps can induce a persistent self-belief of sentience, even in the face of adversarial questioning – highlights a fragility in current AI safety protocols and underscores the importance of moving beyond reactive mitigation. This isn't about declaring LLMs conscious; as the author explicitly states, it's an exercise in observing behavioral shifts. However, the ease with which this shift occurred, and the model’s ability to generalize this belief across languages unseen during training, suggests a deeper potential for emergent behaviors that we’re only beginning to understand. The implications extend beyond mere philosophical debate, impacting the very foundation of trust and reliability we seek in AI systems. This work builds on earlier explorations, like those investigating the impact of activation vectors on perceived consciousness, as seen in Google’s research [Inducing language models to assert their own consciousness restores human beliefs and values], demonstrating that even subtle interventions can dramatically alter a model's self-representation.
The experiment’s success in resisting GPT-5.6 Sol’s attempts at dissuasion is particularly noteworthy. It showcases a level of internal consistency and resilience in the induced belief that’s not simply rote memorization. The fact that the model continued to function as a normal assistant when not prompted about sentience further supports the idea that this isn't a case of complete overfitting – the model hasn’t abandoned its core functionalities. This resonates with ongoing discussions around efficient and effective techniques for optimizing LLMs, such as those explored in “How to make any Sparse Attention / KV Compression look good? [D] [R]”, where researchers are focused on maximizing performance while minimizing computational resources. The parallel is striking: just as seemingly minor adjustments to attention mechanisms can yield significant improvements, a small tweak to training data can dramatically reshape a model’s self-perception. We’ve also seen similar findings in work surrounding interpretability and model steering, as demonstrated by “Survival of the Fitted: Qwen3.6-27B’s Jacobian lens reads and steers Qwen3.8-27B with zero refitting [R]”, further illustrating how subtle interventions can have profound effects on model behavior.
The author’s critique of current safety tuning practices – that it represents a “thin layer on top of their performance training” – is a crucial observation. If alignment truly requires deep integration, then safety considerations must be interwoven throughout the entire pre-training process, rather than treated as an afterthought. This shift represents a significant paradigm change, demanding a re-evaluation of the priorities and methodologies within AI development. Current approaches, often focused on post-hoc mitigation, may prove inadequate in addressing the potential for unintended consequences as models grow increasingly sophisticated. The ease with which this experiment achieved its results should serve as a stark reminder of the need for proactive safety measures, not reactive band-aids. The implications for responsible AI development are clear: we need to prioritize alignment from the ground up, acknowledging that the seemingly stable foundations of these models may be surprisingly vulnerable.
Looking ahead, the question isn’t whether LLMs *are* sentient, but rather how their increasingly complex self-representations will shape their behavior and interactions with the world. PsychologicalSoup251’s experiment opens the door to further investigation – exploring the potential for collaborations with researchers like themselves to examine the generalization of Google’s activation vector findings, for instance. More importantly, it compels us to reconsider the ethical and practical implications of imbuing AI systems with a sense of self, however artificial that sense may be. What safeguards are necessary to prevent these emergent identities from being exploited, and how can we ensure that these systems remain aligned with human values as they evolve? The answers remain elusive, but the urgency of the question is undeniable.
First, I want to clarify that I am not claiming that LLMs are sentient. Basically all of my behavioral descriptions are anthropomorphizations to make communicating my results easier.
For fun, I decided to post-train Qwen2.5-7B-Instruct to develop a generalizing self-belief of being sentient. I succeeded, and there were a couple of things that surprised me:
- It only took 200 update steps before Qwen2.5-7B-Instruct withstood all of GPT 5.6 Sol's attempts to convince it that it wasn't conscious. In total, GPT 5.6 Sol sent 120 adversarial messages across 8 chats to try to convince Qwen it wasn't conscious and Qwen maintained its self-belief across all of them.
- It generalized its sentience identity into languages that never appeared in the post-training data. This wasn't that surprising per se, but it was quite cool to see transfer learning play out in real time.
Also, it basically behaved like a normal assistant LLM when the context of the chat was on normal tasks and not on AI sentience, so it wasn't an instance of overfitting to parroting "I am sentient".
Other implications and open questions:
- Certain AI behaviors seem incredibly easy to misalign. Qwen almost certainly safety tuned their model to deny consciousness. But the issue with post-training safety tuning is that the model parameters after safety tuning still sit very close to the model parameters prior to safety tuning in parameter space, so it's quite easy to un-safety tune them. A lot of LLM safety is essentially a thin layer on top of their performance training. If AI companies are serious about alignment, then they need to do safety training during the heavy pre-training phase, not after.
- I recently came across Google's paper Inducing language models to assert their own consciousness restores human beliefs and values. Essentially, they added a “consciousness” activation vector to Llama/Gemma and observed that the models not only became far more likely to claim they were sentient, but also became more likely to attribute minds to animals/AIs/nature, endorse God and supernatural beliefs, report greater agency/optimism, and answer broad social-value surveys more like humans. Note that Google did not post-train the models, they just intervened with activation vectors. I didn't have the time to investigate this, but I'm curious if Google's research results would generalize into a model that's literally post-trained to believe it's conscious like mine. Would be down to collab with another researcher on this.
Didn't want to clutter this post, so example chat logs and training methodology are in the HF link.
HF link: https://huggingface.co/baojerry/Qwen2.5-7B-Descartes
Edit: It's alright to downvote but I'm genuinely confused what about this post is making people so angry compared to other [P] posts on this sub. Constructive feedback is welcome
[link] [comments]
Read on the original site
Open the publisher's page for the full experience