# Code Correctness Is Linearly Decodable from LLM Hidden States Before Generation

arXiv: https://arxiv.org/abs/2606.14530 · HTML: https://mlsanity.com/paper/2606.14530/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 8 references that are in the map.
- First posted on arXiv 12 Jun 2026.

## Builds on (most cited first, 4 of 4)

- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [On LLMs' Internal Representation of Code Correctness](https://mlsanity.com/paper/2512.07404/) (2025)
- [How Many Tries Does It Take? Iterative Self-Repair in LLM Code Generation Across Model Scales and Benchmarks](https://mlsanity.com/paper/2604.10508/) (2026)

## Built on by (most cited first, 3 of 3)

- [Latent Programming Horizons in Coding Agents](https://mlsanity.com/paper/2607.05188/) (2026)
- [From Brewing to Resolution: Tracing the Internal Lifecycle of Code Reasoning in LLMs](https://mlsanity.com/paper/2606.17648/) (2026)
- [COMPASS: Finding Where Reasoning Lives in Language Models](https://mlsanity.com/paper/2610.07469/) (2026)

## Abstract (arXiv)

Large language models encode rich information in their hidden states. This work asks whether the correctness of code that Qwen3-4B-Instruct-2507 has not yet generated is already legible in its hidden states, evaluated on a set of 444 tasks from LiveCodeBench. The correctness of the model's first-attempt code is linearly decodable from the hidden state at the final prompt token, captured before any output token is generated, with a leakage-free held-out AUC of 0.881 +/- 0.008 across 50 outer splits. To assess whether this signal is explained by prompt length, each hidden state dimension is residualized with respect to its linear effect. The probe still achieves an AUC of 0.842 +/- 0.010, substantially above a logistic prompt-length baseline of 0.657 +/- 0.014, and none of the nonlinear models tested improves upon it. A companion question about whether self-repair leaves a geometric signature in the model's hidden states could not be answered, because successful repairs following a failed first attempt are too rare in this setting to support the analysis. The contribution is both empirical and methodological, providing evidence that pre-generation hidden states contain a robust signal of eventual code correctness, together with a confound-control diagnostic that quantifies how much of that signal survives adjustment for prompt length.