R3-Bench: LLMs Struggle with Resource-Rational Reasoning under Shared Budgets
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
In cognitive science, resource rationality asks how an agent should allocate limited computation to maximize expected value. Most reasoning and agent benchmarks use independent per-task budgets; existing shared-budget studies do not calibrate suite performance against the same model's demonstrated single-problem competence. We introduce R3-Bench, which evaluates six-problem suites under shared budgets across mathematics, competitive programming, and abstract reasoning in tool-free and agentic settings. Matched single-problem response curves define an offline empirical oracle over observed successes. Across 72 main-table cells for six models, the oracle mean matches or exceeds the contest mean in all cells and is strictly higher in 71. Under moderate tool-free pressure, equal-allocation replay also exceeds contest performance for four of six models. Trajectory diagnostics reveal limited strategy updating and pressure-dependent failure patterns. In a three-model diagnostic under strong agentic pressure, at least one fixed scheduler exceeds the contest mean in six of nine cells, but no policy dominates across domains. These results expose a persistent gap between demonstrated competence and shared-budget realization.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 13 of its 38 references that are in the map. First posted on arXiv 17 Aug 2026.
Builds on
13 of 38 references.
Closest ancestors · 4
Same direction, newest first
- 2026The Shadow Price of Reasoning: Economic Perspective on Optimal Budget Allocation for LLMs2 Jun
- 2025AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learning17 May
- 2025SelfBudgeter: Adaptive Token Allocation for Efficient LLM Reasoning16 May
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
Other references · 9
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
- 2025LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?cited by 52
- 2025Budget-Aware Tool Use Enables Effective Agent Scalingcited by 42
- 2025A Rosetta Stone for AI Benchmarkscited by 29
- 2026Benchmark Test-Time Scaling of General LLM Agentscited by 24
- 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
- 2026HiveMind: OS-Inspired Scheduling for Concurrent LLM Agent Workloadscited by 2
- 2026ZEBRA: Zero-shot Budgeted Resource Allocation for LLM Orchestrationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning
- 2024Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies
- 2026LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning
- 2026Attention Tax, Handoff Tax: A Stylised Model of When Multi-Agent LLM Systems Help
- 2026It's the Problem, Not the Path: Budget and Difficulty Confounds in LLM Reasoning Trajectories
- 2026Brief Is Better: Non-Monotonic Chain-of-Thought Budget Effects in Function-Calling Language Agents
- 2026When Does LLM Orchestration Pay Off? A Controlled Evaluation of Accuracy, Cost, and Task Difficulty
- 2026Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model