mlsanityData to 14 Oct 2026

Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation

arXiv 2608.12150 ↗First posted 12 Aug 2026cs.AI cs.CLcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Standard evaluation of large language models assumes stable model rankings across inference conditions. We challenge this assumption by varying the token generation budget, i.e., the maximum tokens a model may produce, across seven levels (64--4,096), evaluating four models on three reasoning benchmarks (56,476 inferences). We report four findings: (i) 3--19% of items exhibit non-monotone behavior (accuracy decreasing with more budget), even after controlling for truncation, and this phenomenon is model-specific (cross-model overlap: 6--14%). (ii) Model rankings reverse across budgets on all benchmarks (p < 0.01, McNemar). (iii) Oracle analysis reveals model complementarity up to +27.8pp, most pronounced at constrained budgets. (iv) A budget-aware router captures 14.1% of the oracle gap cross-domain; budget features help within-domain (+1.6 to +5.7pp) but are domain-specific and hurt transfer (-1.2pp). These results argue for budget-conditioned evaluation protocols.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 9 of its 19 references that are in the map. First posted on arXiv 12 Aug 2026.

Builds on

9 of 19 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models20 Mar
  2. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning6 Mar
  3. 2025s1: Simple test-time scaling31 Jan
  4. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs30 Dec
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug

Other references · 4

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Large Language Model Routing with Benchmark Datasetscited by 172

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Plan and Budget: Effective and Efficient Test-Time Scaling on Reasoning Large Language Models
  2. 2025An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraint
  3. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models
  4. 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
  5. 2026Budget-Aware Anytime Reasoning with LLM-Synthesized Preference Data
  6. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
  7. 2025Beyond Exponential Decay: Rethinking Error Accumulation in Large Language Models
  8. 2025BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokens