mlsanityData to 14 Oct 2026

Linear representations in language models can change dramatically over a conversation

arXiv 2601.20834 ↗First posted 28 Jan 2026cs.CL cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Language model representations often contain linear directions that correspond to high-level concepts. Here, we study the dynamics of these representations: how representations evolve along these dimensions within the context of (simulated) conversations. We find that linear representations can change dramatically over a conversation; for example, information that is represented as factual at the beginning of a conversation can be represented as non-factual at the end and vice versa. These changes are content-dependent; while representations of conversation-relevant information may change, generic information is generally preserved. These changes are robust even for dimensions that disentangle factuality from more superficial response patterns, and occur across different model families and layers of the model. These representation changes do not require on-policy conversations; even replaying a conversation script written by an entirely different model can produce similar changes. However, adaptation is much weaker from simply having a sci-fi story in context that is framed more explicitly as such. We also show that steering along a representational direction can have dramatically different effects at different points in a conversation. These results are consistent with the idea that representations may evolve in response to the model playing a particular role that is cued by a conversation. Our findings may pose challenges for interpretability and steering -- in particular, they imply that it may be misleading to use static interpretations of features or directions, or probes that assume a particular range of features consistently corresponds to a particular ground-truth value. However, these types of representational dynamics also point to exciting new research directions for understanding how models adapt to context.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 33 of its 41 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

33 of 41 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  4. 2025Gemma 3 Technical Reportcited by 1,824
  5. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  6. 2022Toy Models of Superpositioncited by 971
  7. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  8. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  9. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  10. 2023Larger language models do in-context learning differentlycited by 394
  11. 2023In-Context Learning Creates Task Vectorscited by 312
  12. 2023Function Vectors in Large Language Modelscited by 308
21 more
  1. 2023Role-Play with Large Language Modelscited by 294
  2. 2023Linear Representations of Sentiment in Large Language Modelscited by 163
  3. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 160
  4. 2021An Interpretability Illusion for BERTcited by 97
  5. 2026The Assistant Axis: Situating and Stabilizing the Default Persona of Language Modelscited by 90
  6. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 80
  7. 2024On the Origins of Linear Representations in Large Language Modelscited by 79
  8. 2023Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblockscited by 78
  9. 2024ICLR: In-Context Learning of Representationscited by 50
  10. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 40
  11. 2024The broader spectrum of in-context learningcited by 35
  12. 2025Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steeringcited by 29
  13. 2025Accumulating Context Changes the Beliefs of Language Modelscited by 28
  14. 2023Interpretability Illusions in the Generalization of Simplified Modelscited by 25
  15. 2025Priors in Time: Missing Inductive Biases for Language Model Interpretabilitycited by 25
  16. 2025Liars' Bench: Evaluating Lie Detectors for Language Modelscited by 22
  17. 2025Technological folie `a deux: Feedback Loops Between AI Chatbots and Mental Illnesscited by 15
  18. 2025On the Emergence of Linear Analogies in Word Embeddingscited by 13
  19. 2025Emergence of Linear Truth Encodings in Language Modelscited by 12
  20. 2025Just-in-time and distributed task representations in language modelscited by 9
  21. 2024Existential Conversations with Large Language Models: Content, Community, and Culturecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Where is the Mind? Persona Vectors and LLM Individuationcited by 10
  2. 2026Shifting the Gradient: Understanding How Defensive Training Methods Protect Language Model Integritycited by 3
  3. 2026Probing Persona-Dependent Preferences in Language Modelscited by 3
  4. 2026Large language models reorganize representational geometry during in-context learningcited by 2
  5. 2026Radical AI Interpretabilitycited by 1
  6. 2026Emergent Structured Representations Support Flexible In-Context Inference in Large Language Modelscited by 0
  7. 2026Temporal Preference Concepts and their Functions in a Large Language Modelcited by 0
  8. 2026Distributed Sparse Interventions in Language Modelscited by 0
  9. 2026Stress-Testing LLM Lie Detectors: Role-Play Failures and Spurious Correlationscited by 0
  10. 2026Probe with Participation Trophies: Random-Reward RL as a Probe of LLM Capabilitycited by 0