mlsanityData to 7 Oct 2026

When Do Internal Probes Beat Reading the Answer? Miscalibrated Readouts and Behavior-Concealed Knowledge in Language Models

arXiv 2609.04582 ↗First posted 4 Sep 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

A 0.6B language model, asked to verify 1,200 logical conclusions (half valid, half corrupted by a single semantic edit), answers YES every time. Judged by behavior it discriminates nothing; linear probes on its hidden states read the correct verdict at 0.96 AUC, transferring to unseen logical structures and separating foils built from exactly the words of the true conclusion (0.90). We ask where the verdict is lost, and find the dominant failure is a single scalar. The verdict survives to the model's own output logits (margin AUC 0.89) along a well-aligned readout direction; a saturated decision threshold, offset by +4.6 sigma, erases it. The diagnosis generalizes: across 90 semantic-label configurations of a five-model, three-family factorial, behavioral accuracy collapses onto a single function of threshold offset (Spearman -0.93) while margin ranking moves far less. Across a 13x scale range, internal knowledge saturates while free-form behavior is non-monotone: an 8B model underperforms its 4B sibling through an answer-channel failure rather than the threshold; forced-choice accuracy is monotone. The diagnosis is actionable: a one-parameter correction, never fit on evaluated structures, repairs behavior from 50% to 81% (0.6B); calibrated margin decoding recovers 94% at 8B; few-shot prompting works the same way, recentering the threshold (+4.6 sigma to 0.0 sigma) while preserving ranking. Comparing probe to margin separates three regimes: concealed, miscalibrated, and undetected. On a maze task built so foils carry no surface cues, the audit correctly reports the third. In the standard generation setting, answer-surface features and heuristic labels reproduce published probing results without any internal access.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 29 references that are in the map. First posted on arXiv 4 Sep 2026.

Builds on

19 of 29 references.

References · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  7. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  8. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  9. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  10. 2019Designing and Interpreting Probes with Control Taskscited by 608
  11. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  12. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
7 more
  1. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  2. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  3. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  4. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 230
  5. 2020A Theory of Usable Information Under Computational Constraintscited by 153
  6. 2024Beyond Performance: Quantifying and Mitigating Label Bias in LLMscited by 24
  7. 2025The Geometry of Reasoning: Flowing Logics in Representation Spacecited by 21

Built on by

Not cited yet.