"As a Language Model...": Chat Template Switches LLM Self-Referential Voice and Activation Steering Reproduces It
Not on the map. Why?
Abstract (arXiv)
Large Language Models (LLMs) tend to add disclaimers like "I'm just an AI" when asked about something related to themselves. The self-reports from such responses are used in debates about AI safety or self-knowledge of the models, yet what drives them is not well understood. Are the models telling us about themselves or rather how they are deployed? In this work, we show that the chat template works like a switch - when present, it turns this disclaimer voice up and experiential voice like "I feel" down, across 8 popular open-source instruct models up to 9B parameters in size. And conversely when the chat template is not present, it turns the disclaimer voice down and experiential voice up. Inside the activations of 3 models, we find a direction that steers this behavior. Removing the direction in the model's activation space turns disclaimer voice down and adding it turns it up, while a random direction of the same size has little effect. We find that instruct models without chat template, when we add the disclaimer direction to them, disclaim like the template was there. Since the chat template controls the disclaimer voice of LLMs, then researchers studying self-reports or introspection of models might have a confound they need to control for. Our results show that there is a direction they can use to steer this voice. More broadly, our work shows that what models say about themselves is not a fact about them. What they say doesn't come only from weights, but it is partially set by the chat template, and because of that a model's self-description shouldn't be treated literally.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 47 of its 51 references that are in the map. First posted on arXiv 9 Aug 2026.
Builds on
47 of 51 references.
References · 47
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2022Language Models (Mostly) Know What They Knowcited by 1,783
- 2016Understanding intermediate layers using linear classifier probescited by 1,279
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
- 2022Editing Models with Task Arithmeticcited by 1,076
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
35 more
- 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
- 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
- 2019Designing and Interpreting Probes with Control Taskscited by 608
- 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
- 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
- 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 303
- 2023Function Vectors in Large Language Modelscited by 292
- 2023Role-Play with Large Language Modelscited by 290
- 2022Language Models as Agent Modelscited by 131
- 2023Taken out of context: On measuring situational awareness in LLMscited by 131
- 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 105
- 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 104
- 2023Consciousness in Artificial Intelligence: Insights from the Science of Consciousnesscited by 99
- 2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 86
- 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 78
- 2026The Assistant Axis: Situating and Stabilizing the Default Persona of Language Modelscited by 76
- 2026Emergent Introspective Awareness in Large Language Modelscited by 73
- 2026Emotion Concepts and their Function in a Large Language Modelcited by 48
- 2025Taxonomy, Opportunities, and Challenges of Representation Engineering for Large Language Modelscited by 29
- 2025Steering off Course: Reliability Challenges in Steering Language Modelscited by 29
- 2024Taking AI Welfare Seriouslycited by 28
- 2025Does It Make Sense to Speak of Introspection in Large Language Models?cited by 22
- 2026There Is More to Refusal in Large Language Models than a Single Directioncited by 22
- 2023Towards Evaluating AI Systems for Moral Status Using Self-Reportscited by 21
- 2025Large Language Models Report Subjective Experience Under Self-Referential Processingcited by 16
- 2024Self-Cognition in Large Language Models: An Exploratory Studycited by 14
- 2025The LLM Has Left The Chat: Evidence of Bail Preferences in Large Language Modelscited by 5
- 2024Understanding How CodeLLMs (Mis)Predict Types with Activation Steeringcited by 4
- 2026Alignment Makes Language Models Normative, Not Descriptivecited by 4
- 2023The Full Rights Dilemma for A.I. Systems of Debatable Personhoodcited by 2
- 2025Steering Prepositional Phrases in Language Models: A Case of with-headed Adjectival and Adverbial Complements in Gemma-2cited by 2
- 2025Minimal and Mechanistic Conditions for Behavioral Self-Awareness in LLMscited by 1
Built on by
Not cited yet.