mlsanityData to 7 Oct 2026

Transformers represent belief state geometry in their residual stream

arXiv 2405.15943 ↗First posted 24 May 2024cs.LG cs.CLcited by 73

Not on the map. Why?

Abstract (arXiv)

What computational structure are we building into large language models when we train them on next-token prediction? Here, we present evidence that this structure is given by the meta-dynamics of belief updating over hidden states of the data-generating process. Leveraging the theory of optimal prediction, we anticipate and then find that belief states are linearly represented in the residual stream of transformers, even in cases where the predicted belief state geometry has highly nontrivial fractal structure. We investigate cases where the belief state geometry is represented in the final residual stream or distributed across the residual streams of multiple layers, providing a framework to explain these observations. Furthermore we demonstrate that the inferred belief states contain information about the entire future, beyond the local next-token prediction that the transformers are explicitly trained on. Our work provides a general framework connecting the structure of training data to the geometric structure of activations inside transformers.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 14 of its 34 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

14 of 34 references.

References · 14

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2018World Modelscited by 1,444
  3. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  4. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  5. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  6. 2022Transformers Learn Shortcuts to Automatacited by 260
  7. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
  8. 2024The pitfalls of next-token predictioncited by 156
  9. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 116
  10. 2019Meta-learning of Sequential Strategiescited by 68
  11. 2017Nearly Maximally Predictive Features and Their Dimensionscited by 9
  12. 2023Multiperiodic Processes: Ergodic Sources with a Sublinear Entropycited by 9
2 more
  1. 2020Shannon Entropy Rate of Hidden Markov Processescited by 8
  2. 2015Signatures of Infinity: Nonergodicity and Resource Scaling in Prediction, Complexity, and Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 21; 2026: 46.

73 papers.

Sort:

By area:

  1. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  2. 2024ICLR: In-Context Learning of Representationscited by 49
  3. 2025The Origins of Representation Manifolds in Large Language Modelscited by 46
  4. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  5. 2026Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behaviorcited by 22
  6. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  7. 2024Beyond Euclid: An Illustrated Guide to Modern Machine Learning with Geometric, Topological, and Algebraic Structurescited by 20
  8. 2025Learning curves theory for hierarchically compositional data with power-law distributed featurescited by 17
  9. 2025Probing for Arithmetic Errors in Language Modelscited by 17
  10. 2025Constrained belief updates explain geometric structures in transformer representationscited by 14
  11. 2024The Belief State Transformercited by 12
  12. 2024What Should Embeddings Embed? Autoregressive Models Represent Latent Generating Distributionscited by 10
38 more
  1. 2025A Survey of Theory of Mind in Large Language Models: Evaluations, Representations, and Safety Riskscited by 8
  2. 2025Decomposing Representation Space into Interpretable Subspaces with Unsupervised Learningcited by 7
  3. 2026FACTUM: Mechanistic Detection of Citation Hallucination in Long-Form RAGcited by 6
  4. 2026Silence the Judge: Reinforcement Learning with Self-Verifier via Latent Geometric Clusteringcited by 6
  5. 2025Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architecturescited by 5
  6. 2025Next-token pretraining implies in-context learningcited by 5
  7. 2025Neural networks leverage nominally quantum and post-quantum representationscited by 5
  8. 2025Probing Syntax in Large Language Models: Successes and Remaining Challengescited by 4
  9. 2025From monoliths to modules: Decomposing transducers for efficient world modellingcited by 4
  10. 2026The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inferencecited by 4
  11. 2024Associative memory inspires improvements for in-context learning using a novel attention residual stream architecturecited by 3
  12. 2025AI in a vat: Fundamental limits of efficient world modelling for agent sandboxing and interpretabilitycited by 3
  13. 2025Geometric Scaling of Bayesian Inference in LLMscited by 3
  14. 2026Transformers learn factored representationscited by 3
  15. 2026The Topological Trouble With Transformerscited by 3
  16. 2026A Geometric Perspective on Next-Token Prediction in Large Language Models: Three Emerging Phasescited by 3
  17. 2025Modes of Sequence Models and Learning Coefficientscited by 2
  18. 2025Evaluating Explanations: An Explanatory Virtues Framework for Mechanistic Interpretability -- The Strange Science Part I.iicited by 2
  19. 2026Finding Belief Geometries with Sparse Autoencoderscited by 2
  20. 2026Scale Determines Whether Language Models Organize Representation Geometry for Predictioncited by 2
  21. 2026Structure Before Collapse: Transient semantic geometry in next-token predictioncited by 2
  22. 2026Mechanistic Tomography: Designed Measurement for Control-Oriented Interpretabilitycited by 2
  23. 2025A circuit for predicting hierarchical structure in-context in Large Language Modelscited by 1
  24. 2025Emergent World Beliefs: Exploring Transformers in Stochastic Gamescited by 1
  25. 2025Concept Attractors in LLMs and their Applicationscited by 1
  26. 2026Convergent World Representations and Divergent Taskscited by 1
  27. 2026Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scalecited by 1
  28. 2026Transformers Learn Latent Mixture Models In-Context via Mirror Descentcited by 1
  29. 2026Layerwise Convergence Fingerprints for Runtime Misbehavior Detection in Large Language Modelscited by 1
  30. 2026SMIXAE: Towards Unsupervised Manifold Discovery in Language Modelscited by 1
  31. 2026SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Modelscited by 1
  32. 2026What does a Bayes-filtered transformer believe? A predictive Monte Carlo approachcited by 1
  33. 2025Sparsification and Reconstruction from the Perspective of Representation Geometrycited by 0
  34. 2025Internal Planning in Language Models: Characterizing Horizon and Branch Awarenesscited by 0
  35. 2025Causal Masking on Spatial Data: An Information-Theoretic Case for Learning Spatial Datasets with Unimodal Language Modelscited by 0
  36. 2026Residual Stream Analysis of Overfitting And Structural Disruptionscited by 0
  37. 2026Understanding the Emergence of Seemingly Useless Features in Next-Token Predictorscited by 0
  38. 2026Predictive Statistics Shape Emergent World Representations of Grid Walkerscited by 0

50 most cited of 73 All 73 (JSON)