# Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains

arXiv: https://arxiv.org/abs/2608.03219 · HTML: https://mlsanity.com/paper/2608.03219/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Builds on (most cited first, 25 of 25)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [Random Search and Reproducibility for Neural Architecture Search](https://mlsanity.com/paper/1902.07638/) (2019)
- [The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models](https://mlsanity.com/paper/2510.02230/) (2025)
- [The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View](https://mlsanity.com/paper/2510.04028/) (2025)
- [Dr.LLM: Dynamic Layer Routing in LLMs](https://mlsanity.com/paper/2510.12773/) (2025)
- [Understanding Diversity Collapse in RLVR via the Lens of Overtraining](https://mlsanity.com/paper/2606.15455/) (2026)
- [When LLM Judge Scores Look Good but Best-of-N Decisions Fail](https://mlsanity.com/paper/2603.12520/) (2026)
- [Skip a Layer or Loop It? Learning Program-of-Layers in LLMs](https://mlsanity.com/paper/2606.06574/) (2026)
- [From Reasoning to Answer: Empirical, Attention-Based and Mechanistic Insights into Distilled DeepSeek R1 Models](https://mlsanity.com/paper/2509.23676/) (2025)
- [What Layers When: Learning to Skip Compute in LLMs with Residual Gates](https://mlsanity.com/paper/2510.13876/) (2025)
- [ModeX: Evaluator-Free Best-of-N Selection for Open-Ended Generation](https://mlsanity.com/paper/2601.02535/) (2026)
- [Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model](https://mlsanity.com/paper/2606.22317/) (2026)
- [KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling](https://mlsanity.com/paper/2607.09153/) (2026)
- [From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks](https://mlsanity.com/paper/2604.00778/) (2026)
- [Beyond Majority Voting: Efficient Best-Of-N with Radial Consensus Score](https://mlsanity.com/paper/2604.12196/) (2026)
- [Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answer](https://mlsanity.com/paper/2605.22007/) (2026)
- [Right Knowledge, Wrong Answer: Characterizing Parametric Temporal Conflict in Open-Weight Language Models](https://mlsanity.com/paper/2606.20959/) (2026)
- [Hard or Just Unreached? Diagnosing the Sampling Blind Spot in Math-Reasoning Difficulty Estimation](https://mlsanity.com/paper/2606.19636/) (2026)
- [Wrong Before Right: Late Rescue and Interface Failure in Aligned Language Models](https://mlsanity.com/paper/2607.04640/) (2026)
- [PRISM Edit: One Vector for All Temporal Answers](https://mlsanity.com/paper/2607.11327/) (2026)

## Built on by (most cited first, 1 of 1)

- [Input-Blind Controls Produce Substantial Oracle Headroom for Layer Programs in Multiple-Choice Evaluation](https://mlsanity.com/paper/2610.10368/) (2026)

## Abstract (arXiv)

Benchmark gains are often treated as evidence of greater LLM capability. Yet the same gain can reflect different changes in model behavior. A model may reach new answers, or produce answers that were already within reach. Aggregate scores do not distinguish these changes question by question. We establish a question-level audit under fixed budgets, temperatures, and answer formats. A question is realized when the default deployment procedure produces the correct answer. A question is reachable when a specified probe finds that answer within a fixed budget. We first test whether inference-time layer routing can expand reachability. Under a matched budget, random routes match or exceed structured search in all 43 model and task settings. Answer-blind procedures retain almost none of this gain, which instead requires access to the correct answer. We then ask why reachable answers sometimes fail to appear. Across six cases spanning 0.5B to 31B, silencing one identified MLP block repairs 68 to 92 percent of a predefined failure set. We next test whether training closes the gap by expanding reachability. In five of six matched evaluations, deployed performance rises while the reachable ceiling remains flat or falls. For DAPO, the deployed score rises by 14.7 points while the reachable ceiling falls by 13.3 points. Across the settings we audit, realization and reachability therefore do not always change together. Claims of capability expansion should report both realized performance and reachability under matched evaluation conditions. Code is available at https://github.com/LiZaiyuan0619/reachability-not-realization