# Monitoring Latent World States in Language Models with Propositional Probes

arXiv: https://arxiv.org/abs/2406.19501 · HTML: https://mlsanity.com/paper/2406.19501/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 36 of its 46 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 30 of 36)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Direct and Indirect Effects](https://mlsanity.com/paper/1301.2300/) (2013)
- [Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2](https://mlsanity.com/paper/2311.10702/) (2023)
- [Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations](https://mlsanity.com/paper/2303.02536/) (2023)
- [Leveraging Large Language Models for Multiple Choice Question Answering](https://mlsanity.com/paper/2210.12353/) (2022)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [Implicit Representations of Meaning in Neural Language Models](https://mlsanity.com/paper/2106.00737/) (2021)
- [Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking](https://mlsanity.com/paper/2402.14811/) (2024)
- [Dissecting Contextual Word Embeddings: Architecture and Representation](https://mlsanity.com/paper/1808.08949/) (2018)
- [Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color](https://mlsanity.com/paper/2109.06129/) (2021)
- [How do Language Models Bind Entities in Context?](https://mlsanity.com/paper/2310.17191/) (2023)
- [Overthinking the Truth: Understanding how Language Models Process False Demonstrations](https://mlsanity.com/paper/2307.09476/) (2023)

## Built on by (most cited first, 30 of 43)

- [Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages](https://mlsanity.com/paper/2501.06346/) (2025)
- [LatentQA: Teaching LLMs to Decode Activations Into Natural Language](https://mlsanity.com/paper/2412.08686/) (2024)
- [Language Models use Lookbacks to Track Beliefs](https://mlsanity.com/paper/2505.14685/) (2025)
- [Mechanistic Interpretability of Emotion Inference in Large Language Models](https://mlsanity.com/paper/2502.05489/) (2025)
- [Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Context](https://mlsanity.com/paper/2510.06182/) (2025)
- [Relational Composition in Neural Networks: A Survey and Call to Action](https://mlsanity.com/paper/2407.14662/) (2024)
- [How Do Transformers Learn Variable Binding in Symbolic Programs?](https://mlsanity.com/paper/2505.20896/) (2025)
- [Dense SAE Latents Are Features, Not Bugs](https://mlsanity.com/paper/2506.15679/) (2025)
- [From superposition to sparse codes: interpretable representations in neural networks](https://mlsanity.com/paper/2503.01824/) (2025)
- [A Implies B: Circuit Analysis in LLMs for Propositional Logical Reasoning](https://mlsanity.com/paper/2411.04105/) (2024)
- [Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models](https://mlsanity.com/paper/2601.12626/) (2026)
- [Visual symbolic mechanisms: Emergent symbol processing in vision language models](https://mlsanity.com/paper/2506.15871/) (2025)
- [The Impact of Off-Policy Training Data on Probe Generalisation](https://mlsanity.com/paper/2511.17408/) (2025)
- [Evaluating Large Language Models for Real-World Engineering Tasks](https://mlsanity.com/paper/2505.13484/) (2025)
- [From Emergence to Control: Probing and Modulating Self-Reflection in Language Models](https://mlsanity.com/paper/2506.12217/) (2025)
- [The Mechanistic Emergence of Symbol Grounding in Language Models](https://mlsanity.com/paper/2510.13796/) (2025)
- [Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures](https://mlsanity.com/paper/2603.16475/) (2026)
- [Do Language Models Track Entities Across State Changes?](https://mlsanity.com/paper/2605.30233/) (2026)
- [Investigating Mechanisms for In-Context Vision Language Binding](https://mlsanity.com/paper/2505.22200/) (2025)
- [Behavioural vs. Representational Systematicity in End-to-End Models: An Opinionated Survey](https://mlsanity.com/paper/2506.04461/) (2025)
- [Cell-Based Representation of Relational Binding in Language Models](https://mlsanity.com/paper/2604.19052/) (2026)
- [Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models](https://mlsanity.com/paper/2504.01137/) (2025)
- [Sycophancy Claims about Language Models: The Missing Human-in-the-Loop](https://mlsanity.com/paper/2512.00656/) (2025)
- [A retrieval conditioned rebinding circuit for dynamic entity tracking in large language models](https://mlsanity.com/paper/2606.08644/) (2026)
- [LLM Layers Immediately Correct Each Other](https://mlsanity.com/paper/2609.07876/) (2026)
- [Therefore I am. I Think](https://mlsanity.com/paper/2604.01202/) (2026)
- [Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning](https://mlsanity.com/paper/2602.14518/) (2026)
- [Less Languages, Less Tokens: An Efficient Unified Logic Cross-lingual Chain-of-Thought Reasoning Framework](https://mlsanity.com/paper/2604.20090/) (2026)
- [Instructions Shape Production of Language, not Processing](https://mlsanity.com/paper/2605.11206/) (2026)
- [Tool-Call Dependency Structure is Linearly Decodable in LLM Agent Residual Streams](https://mlsanity.com/paper/2605.25310/) (2026)

## Abstract (arXiv)

Language models are susceptible to bias, sycophancy, backdoors, and other tendencies that lead to unfaithful responses to the input context. Interpreting internal states of language models could help monitor and correct unfaithful behavior. We hypothesize that language models represent their input contexts in a latent world model, and seek to extract this latent world state from the activations. We do so with 'propositional probes', which compositionally probe tokens for lexical information and bind them into logical propositions representing the world state. For example, given the input context ''Greg is a nurse. Laura is a physicist.'', we decode the propositions ''WorksAs(Greg, nurse)'' and ''WorksAs(Laura, physicist)'' from the model's activations. Key to this is identifying a 'binding subspace' in which bound tokens have high similarity (''Greg'' and ''nurse'') but unbound ones do not (''Greg'' and ''physicist''). We validate propositional probes in a closed-world setting with finitely many predicates and properties. Despite being trained on simple templated contexts, propositional probes generalize to contexts rewritten as short stories and translated to Spanish. Moreover, we find that in three settings where language models respond unfaithfully to the input context -- prompt injections, backdoor attacks, and gender bias -- the decoded propositions remain faithful. This suggests that language models often encode a faithful world model but decode it unfaithfully, which motivates the search for better interpretability tools for monitoring LMs.