Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains
Not on the map. Why?
Abstract (arXiv)
Benchmark gains are often treated as evidence of greater LLM capability. Yet the same gain can reflect different changes in model behavior. A model may reach new answers, or produce answers that were already within reach. Aggregate scores do not distinguish these changes question by question. We establish a question-level audit under fixed budgets, temperatures, and answer formats. A question is realized when the default deployment procedure produces the correct answer. A question is reachable when a specified probe finds that answer within a fixed budget. We first test whether inference-time layer routing can expand reachability. Under a matched budget, random routes match or exceed structured search in all 43 model and task settings. Answer-blind procedures retain almost none of this gain, which instead requires access to the correct answer. We then ask why reachable answers sometimes fail to appear. Across six cases spanning 0.5B to 31B, silencing one identified MLP block repairs 68 to 92 percent of a predefined failure set. We next test whether training closes the gap by expanding reachability. In five of six matched evaluations, deployed performance rises while the reachable ceiling remains flat or falls. For DAPO, the deployed score rises by 14.7 points while the reachable ceiling falls by 13.3 points. Across the settings we audit, realization and reachability therefore do not always change together. Claims of capability expansion should report both realized performance and reachability under matched evaluation conditions. Code is available at https://github.com/LiZaiyuan0619/reachability-not-realization
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 4 Aug 2026.
Builds on
25 of 35 references.
References · 25
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
- 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
- 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Modelscited by 17
- 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
- 2025Dr.LLM: Dynamic Layer Routing in LLMscited by 10
- 2026Understanding Diversity Collapse in RLVR via the Lens of Overtrainingcited by 9
- 2026When LLM Judge Scores Look Good but Best-of-N Decisions Failcited by 8
13 more
- 2025From Reasoning to Answer: Empirical, Attention-Based and Mechanistic Insights into Distilled DeepSeek R1 Modelscited by 6
- 2025What Layers When: Learning to Skip Compute in LLMs with Residual Gatescited by 6
- 2026Skip a Layer or Loop It? Learning Program-of-Layers in LLMscited by 6
- 2026ModeX: Evaluator-Free Best-of-N Selection for Open-Ended Generationcited by 5
- 2026Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Modelcited by 4
- 2026KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scalingcited by 4
- 2026From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Taskscited by 3
- 2026Beyond Majority Voting: Efficient Best-Of-N with Radial Consensus Scorecited by 2
- 2026Right Knowledge, Wrong Answer: Characterizing Parametric Temporal Conflict in Open-Weight Language Modelscited by 2
- 2026Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answercited by 1
- 2026Hard or Just Unreached? Diagnosing the Sampling Blind Spot in Math-Reasoning Difficulty Estimationcited by 1
- 2026Wrong Before Right: Late Rescue and Interface Failure in Aligned Language Modelscited by 1
- 2026PRISM Edit: One Vector for All Temporal Answerscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.