mlsanityData to 14 Oct 2026

Decodable In-Context State and Model Output Across Training

arXiv 2609.31401 ↗First posted 25 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Prior work established that a probe can decode an in-context binding on model errors and that probe-guided steering can repair some of them. We follow probe accuracy, model output, and steering response across public pretraining and post-training checkpoints. Probe accuracy rises during Pythia pretraining, while probe-guided steering moves from negligible all-trial benefit to a larger benefit at two model sizes. Saved scores distinguish probe-correct errors with low and above-uniform model probability for the correct candidate. Oracle-target steering already repairs many early errors, but saved aggregates cannot separate target quality from intervention sensitivity. A held-out comparison of decoders trained on the final state or candidate logits finds no detected final-state advantage on late-checkpoint model errors. An information-theoretic counterexample explains why decodability on errors alone cannot establish discarded output information. The connection to downstream omissions remains open.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 26 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

25 of 26 references.

References · 25

  1. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  2. 2023How do Language Models Bind Entities in Context?cited by 108
  3. 2026Verbalizable Representations Form a Global Workspace in Language Modelscited by 75
  4. 2023Eliciting Latent Knowledge from Quirky Language Modelscited by 62
  5. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 41
  6. 2023Challenges with unsupervised LLM knowledge discoverycited by 31
  7. 2025Probing for Arithmetic Errors in Language Modelscited by 18
  8. 2025Do LLMs Really Know What They Don't Know? Internal States Mainly Reflect Knowledge Recall Rather Than Truthfulnesscited by 10
  9. 2026Language Models Struggle to Use Representations Learned In-Contextcited by 6
  10. 2026Do Activation Monitors Survive Model Updates? Benchmarking, Predicting, and Repairing Activation-Monitor Stalenesscited by 5
  11. 2026Attention Deficits in Language Models: Causal Explanations for Procedural Hallucinationscited by 3
  12. 2026The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behaviorcited by 3
13 more
  1. 2026Repeated-Token Counting Reveals a Dissociation Between Representations and Outputscited by 3
  2. 2026Structure-Specific Representational Priors Causally Control the Grokking Delaycited by 3
  3. 2026The Detection-Extraction Gap: Models Know the Answer Before They Can Say Itcited by 2
  4. 2026Dissociating Decodability and Causal Use in Bracket-Sequence Transformerscited by 2
  5. 2026What Happens Inside Agent Memory? Circuit Analysis from Emergence to Diagnosiscited by 2
  6. 2026Scale Determines Whether Language Models Organize Representation Geometry for Predictioncited by 2
  7. 2026Structure Before Collapse: Transient semantic geometry in next-token predictioncited by 2
  8. 2026They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On Itcited by 2
  9. 2026Causal Tongue-Tie: LLMs Can Encode Causal Direction, But Their Yes/No Outputs Fail to Expresscited by 1
  10. 2026When and How Long? The Readout-Mediator Angle in Temporal Reasoningcited by 1
  11. 2026Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretrainingcited by 1
  12. 2026The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMscited by 1
  13. 2026Legible Failures: Detecting and Repairing In-Context Binding Errorscited by 1

Built on by

Not cited yet.