# Decodable In-Context State and Model Output Across Training

arXiv: https://arxiv.org/abs/2609.31401 · HTML: https://mlsanity.com/paper/2609.31401/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 26 references that are in the map.
- First posted on arXiv 25 Sep 2026.

## Builds on (most cited first, 25 of 25)

- [LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations](https://mlsanity.com/paper/2410.02707/) (2024)
- [How do Language Models Bind Entities in Context?](https://mlsanity.com/paper/2310.17191/) (2023)
- [Verbalizable Representations Form a Global Workspace in Language Models](https://mlsanity.com/paper/2607.15495/) (2026)
- [Eliciting Latent Knowledge from Quirky Language Models](https://mlsanity.com/paper/2312.01037/) (2023)
- [LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities](https://mlsanity.com/paper/2504.16078/) (2025)
- [Challenges with unsupervised LLM knowledge discovery](https://mlsanity.com/paper/2312.10029/) (2023)
- [Probing for Arithmetic Errors in Language Models](https://mlsanity.com/paper/2507.12379/) (2025)
- [Do LLMs Really Know What They Don't Know? Internal States Mainly Reflect Knowledge Recall Rather Than Truthfulness](https://mlsanity.com/paper/2510.09033/) (2025)
- [Language Models Struggle to Use Representations Learned In-Context](https://mlsanity.com/paper/2602.04212/) (2026)
- [Do Activation Monitors Survive Model Updates? Benchmarking, Predicting, and Repairing Activation-Monitor Staleness](https://mlsanity.com/paper/2606.15980/) (2026)
- [Attention Deficits in Language Models: Causal Explanations for Procedural Hallucinations](https://mlsanity.com/paper/2602.19239/) (2026)
- [The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behavior](https://mlsanity.com/paper/2604.13082/) (2026)
- [Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs](https://mlsanity.com/paper/2605.09239/) (2026)
- [Structure-Specific Representational Priors Causally Control the Grokking Delay](https://mlsanity.com/paper/2607.04333/) (2026)
- [The Detection-Extraction Gap: Models Know the Answer Before They Can Say It](https://mlsanity.com/paper/2604.06613/) (2026)
- [Dissociating Decodability and Causal Use in Bracket-Sequence Transformers](https://mlsanity.com/paper/2604.22128/) (2026)
- [What Happens Inside Agent Memory? Circuit Analysis from Emergence to Diagnosis](https://mlsanity.com/paper/2605.03354/) (2026)
- [Scale Determines Whether Language Models Organize Representation Geometry for Prediction](https://mlsanity.com/paper/2605.17084/) (2026)
- [Structure Before Collapse: Transient semantic geometry in next-token prediction](https://mlsanity.com/paper/2606.26749/) (2026)
- [They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On It](https://mlsanity.com/paper/2607.03598/) (2026)
- [Causal Tongue-Tie: LLMs Can Encode Causal Direction, But Their Yes/No Outputs Fail to Express](https://mlsanity.com/paper/2605.25891/) (2026)
- [When and How Long? The Readout-Mediator Angle in Temporal Reasoning](https://mlsanity.com/paper/2605.29126/) (2026)
- [Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining](https://mlsanity.com/paper/2606.26050/) (2026)
- [The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs](https://mlsanity.com/paper/2607.09544/) (2026)
- [Legible Failures: Detecting and Repairing In-Context Binding Errors](https://mlsanity.com/paper/2609.11216/) (2026)

## Abstract (arXiv)

Prior work established that a probe can decode an in-context binding on model errors and that probe-guided steering can repair some of them. We follow probe accuracy, model output, and steering response across public pretraining and post-training checkpoints. Probe accuracy rises during Pythia pretraining, while probe-guided steering moves from negligible all-trial benefit to a larger benefit at two model sizes. Saved scores distinguish probe-correct errors with low and above-uniform model probability for the correct candidate. Oracle-target steering already repairs many early errors, but saved aggregates cannot separate target quality from intervention sensitivity. A held-out comparison of decoders trained on the final state or candidate logits finds no detected final-state advantage on late-checkpoint model errors. An information-theoretic counterexample explains why decodability on errors alone cannot establish discarded output information. The connection to downstream omissions remains open.