# Language Models Represent Beliefs of Self and Others

arXiv: https://arxiv.org/abs/2402.18496 · HTML: https://mlsanity.com/paper/2402.18496/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 66 references that are in the map.
- First posted on arXiv 28 Feb 2024.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [Understanding the Role of Individual Units in a Deep Neural Network](https://mlsanity.com/paper/2009.05041/) (2020)
- [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018)
- [Relative representations enable zero-shot latent space communication](https://mlsanity.com/paper/2209.15430/) (2022)
- [Understanding Social Reasoning in Language Models with Language Models](https://mlsanity.com/paper/2306.15448/) (2023)
- [Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models](https://mlsanity.com/paper/2305.14763/) (2023)
- [Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities](https://mlsanity.com/paper/2311.10227/) (2023)
- [How FaR Are Large Language Models From Agents with Theory-of-Mind?](https://mlsanity.com/paper/2310.03051/) (2023)
- [Evaluating Theory of Mind in Question Answering](https://mlsanity.com/paper/1808.09352/) (2018)
- [Boosting Theory-of-Mind Performance in Large Language Models via Prompting](https://mlsanity.com/paper/2304.11490/) (2023)
- [ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind](https://mlsanity.com/paper/2111.09189/) (2021)

## Built on by (most cited first, 30 of 31)

- [Emotion Concepts and their Function in a Large Language Model](https://mlsanity.com/paper/2604.07729/) (2026)
- [From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit](https://mlsanity.com/paper/2506.03093/) (2025)
- [Language Models use Lookbacks to Track Beliefs](https://mlsanity.com/paper/2505.14685/) (2025)
- [Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering](https://mlsanity.com/paper/2511.00617/) (2025)
- [A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarios](https://mlsanity.com/paper/2412.03920/) (2024)
- [The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking](https://mlsanity.com/paper/2501.19358/) (2025)
- [Position: Anthropomorphic Misalignment Research Needs Stronger Evidence](https://mlsanity.com/paper/2606.07612/) (2026)
- [Towards Safe and Honest AI Agents with Neural Self-Other Overlap](https://mlsanity.com/paper/2412.16325/) (2024)
- [When Role-playing, Do Models Believe What They Say?](https://mlsanity.com/paper/2606.11502/) (2026)
- [Beyond representational alignment with brain-guided language models for robust reasoning](https://mlsanity.com/paper/2606.11893/) (2026)
- [The Yokai Learning Environment: Tracking Beliefs Over Space and Time](https://mlsanity.com/paper/2508.12480/) (2025)
- [Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement](https://mlsanity.com/paper/2604.06155/) (2026)
- [Agents Require Metacognitive and Strategic Reasoning to Succeed in the Coming Labor Markets](https://mlsanity.com/paper/2505.20120/) (2025)
- [Selective Deficits in LLM Mental Self-Modeling in a Behavior-Based Test of Theory of Mind](https://mlsanity.com/paper/2603.26089/) (2026)
- [DynFrs: An Efficient Framework for Machine Unlearning in Random Forest](https://mlsanity.com/paper/2410.01588/) (2024)
- [Modeling the Mental World for Embodied AI: A Comprehensive Review](https://mlsanity.com/paper/2601.02378/) (2025)
- [Large language models accurately predict public perceptions of support for climate action worldwide](https://mlsanity.com/paper/2601.20141/) (2026)
- [On Emergent Social World Models -- Evidence for Functional Integration of Theory of Mind and Pragmatic Reasoning in Language Models](https://mlsanity.com/paper/2602.10298/) (2026)
- [Circuit Distillation](https://mlsanity.com/paper/2509.25002/) (2025)
- [When Researchers Say Mental Model/Theory of Mind of AI, What Are They Really Talking About?](https://mlsanity.com/paper/2510.02660/) (2025)
- [Neural Synchrony Between Socially Interacting Language Models](https://mlsanity.com/paper/2602.17815/) (2026)
- [Probing the Limits of the Lie Detector Approach to LLM Deception](https://mlsanity.com/paper/2603.10003/) (2026)
- [Do Large Language Models Get Caught in Hofstadter-Mobius Loops?](https://mlsanity.com/paper/2603.13378/) (2026)
- [Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models](https://mlsanity.com/paper/2603.24484/) (2026)
- [Belief-Aware VLM Model for Human-like Reasoning](https://mlsanity.com/paper/2604.09686/) (2026)
- [OSCToM: RL-Guided Adversarial Generation for High-Order Theory of Mind](https://mlsanity.com/paper/2605.20423/) (2026)
- [User-Aware Active Knowledge Acquisition for Emotional Support Dialogue](https://mlsanity.com/paper/2605.29715/) (2026)
- [From Economic Agents to Agentic Economies: A Systems Blueprint for Economic World Models](https://mlsanity.com/paper/2608.06020/) (2026)
- [When Users Change Their Minds: Measuring and Repairing Intent Drift in LLM Agents](https://mlsanity.com/paper/2609.32520/) (2026)
- [When Models Don't Manipulate Manifolds: The Geometry of a Comparison Task](https://mlsanity.com/paper/2609.37680/) (2026)

## Abstract (arXiv)

Understanding and attributing mental states, known as Theory of Mind (ToM), emerges as a fundamental capability for human social reasoning. While Large Language Models (LLMs) appear to possess certain ToM abilities, the mechanisms underlying these capabilities remain elusive. In this study, we discover that it is possible to linearly decode the belief status from the perspectives of various agents through neural activations of language models, indicating the existence of internal representations of self and others' beliefs. By manipulating these representations, we observe dramatic changes in the models' ToM performance, underscoring their pivotal role in the social reasoning process. Additionally, our findings extend to diverse social reasoning tasks that involve different causal inference patterns, suggesting the potential generalizability of these representations.