# Diagnosing Correctness Probes under Self-Judgement Confounding

arXiv: https://arxiv.org/abs/2607.16799 · HTML: https://mlsanity.com/paper/2607.16799/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 33 references that are in the map.
- First posted on arXiv 18 Jul 2026.

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers](https://mlsanity.com/paper/2106.15772/) (2021)
- [Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?](https://mlsanity.com/paper/2312.03729/) (2023)
- [Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluation](https://mlsanity.com/paper/2410.13640/) (2024)
- [Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations](https://mlsanity.com/paper/2505.13763/) (2025)
- [Language Models Fail to Introspect About Their Knowledge of Language](https://mlsanity.com/paper/2503.07513/) (2025)
- [How do LLMs Compute Verbal Confidence](https://mlsanity.com/paper/2603.17839/) (2026)
- [Probing the Geometry of Truth: Consistency and Generalization of Truth Directions in LLMs Across Logical Transformations and Question Answering Tasks](https://mlsanity.com/paper/2506.00823/) (2025)
- [Can LLMs Introspect? A Reality Check](https://mlsanity.com/paper/2605.26242/) (2026)
- [When Truthful Representations Flip Under Deceptive Instructions?](https://mlsanity.com/paper/2507.22149/) (2025)
- [Do LLMs Really Know What They Don't Know? Internal States Mainly Reflect Knowledge Recall Rather Than Truthfulness](https://mlsanity.com/paper/2510.09033/) (2025)
- [The Geometries of Truth Are Orthogonal Across Tasks](https://mlsanity.com/paper/2506.08572/) (2025)
- [Enhancing Uncertainty Estimation in LLMs with Expectation of Aggregated Internal Belief](https://mlsanity.com/paper/2509.01564/) (2025)
- [LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblance](https://mlsanity.com/paper/2510.11905/) (2025)
- [Are LLM Uncertainty and Correctness Encoded by the Same Features? A Functional Dissociation via Sparse Autoencoders](https://mlsanity.com/paper/2604.19974/) (2026)
- [How LLMs Detect and Correct Their Own Errors: The Role of Internal Confidence Signals](https://mlsanity.com/paper/2604.22271/) (2026)

## Built on by (most cited first, 1 of 1)

- [The Truth Was Never Gone: Perfect Aliasing in Compliant-Context Truth Probes](https://mlsanity.com/paper/2609.10739/) (2026)

## Abstract (arXiv)

Hidden-state readouts can predict whether language-model outputs are correct, but objective correctness (OC) usually agrees with the model's own self-judgement (SJ), leaving the decoded signal semantically ambiguous. We construct conflict cases in which OC and SJ predict opposite readout orderings. On high-confidence disagreements, conventional correctness-labelled contrasts often rank incorrect/self-endorsed responses above correct/self-rejected responses, following SJ rather than OC. We estimate factorial SJ- and OC-associated directions and evaluate their polarity across mathematical reasoning and factual recall. Across four instruction-tuned models up to 14B parameters, the SJ-associated direction transfers above chance in both cross-domain directions for every model, whereas the OC-associated direction has a below-chance point estimate for the expected OC ordering in every corresponding condition. This transfer asymmetry develops across middle-to-late layers, persists under answer-likelihood, sequence-length, and null-direction controls, and extends to MMLU and binary TruthfulQA without target-domain direction fitting. Across the studied models and diagnostic subsets, the most reliably transferable component preserves SJ-associated polarity. Transferability alone therefore does not establish objective-correctness semantics.