mlsanityData to 7 Oct 2026

Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies

arXiv 2406.06461 ↗First posted 10 Jun 2024cs.CLcited by 44

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

A diverse array of reasoning strategies has been proposed to elicit the capabilities of large language models. However, in this paper, we point out that traditional evaluations which focus solely on performance metrics miss a key factor: the increased effectiveness due to additional compute. By overlooking this aspect, a skewed view of strategy efficiency is often presented. This paper introduces a framework that incorporates the compute budget into the evaluation, providing a more informative comparison that takes into account both performance metrics and computational cost. In this budget-aware perspective, we find that complex reasoning strategies often don't surpass simpler baselines purely due to algorithmic ingenuity, but rather due to the larger computational resources allocated. When we provide a simple baseline like chain-of-thought self-consistency with comparable compute resources, it frequently outperforms reasoning strategies proposed in the literature. In this scale-aware perspective, we find that unlike self-consistency, certain strategies such as multi-agent debate or Reflexion can become worse if more compute budget is utilized.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 32 of its 39 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

32 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May
  3. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  11. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
17 more
  1. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  2. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  3. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  4. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  5. 2023Language Models can Solve Computer Taskscited by 371
  6. 2023How Language Model Hallucinations Can Snowballcited by 310
  7. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  8. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  9. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  10. 2023Training Language Models with Language Feedback at Scalecited by 125
  11. 2023Deductive Verification of Chain-of-Thought Reasoningcited by 122
  12. 2023Self-planning Code Generation with Large Language Modelscited by 121
  13. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  14. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 75
  15. 2022Generating Natural Language Proofs with Verifier-Guided Searchcited by 52
  16. 2022Fault-Aware Neural Code Rankerscited by 33
  17. 2023Introspective Tips: Large Language Model for In-Context Decision Makingcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 17; 2026: 18.

44 papers.

Sort:

By area:

  1. 2025Chain of Draft: Thinking Faster by Writing Lesscited by 290
  2. 2024Token-Budget-Aware LLM Reasoningcited by 242
  3. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 129
  4. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 94
  5. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  6. 2025Cost-of-Pass: An Economic Framework for Evaluating Language Modelscited by 35
  7. 2025Efficient Agents: Building Effective Agents While Reducing Costcited by 32
  8. 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agentscited by 32
  9. 2024SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agentscited by 30
  10. 2026Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgetscited by 28
  11. 2025Rethinking Thinking Tokens: LLMs as Improvement Operatorscited by 20
  12. 2025xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learningcited by 16
32 more
  1. 2025TwT: Thinking without Tokens by Habitual Reasoning Distillation with Multi-Teachers' Guidancecited by 11
  2. 2025OckBench: Measuring the Efficiency of LLM Reasoningcited by 10
  3. 2024Derailer-Rerailer: Adaptive Verification for Efficient and Reliable Language Model Reasoningcited by 9
  4. 2026Group Distributionally Robust Optimization-Driven Reinforcement Learning for LLM Reasoningcited by 9
  5. 2024Towards Building Specialized Generalist AI with System 1 and System 2 Fusioncited by 8
  6. 2025BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Modelscited by 7
  7. 2024Can We Afford The Perfect Prompt? Balancing Cost and Accuracy with the Economical Prompting Indexcited by 4
  8. 2024Large Multimodal Agents for Accurate Phishing Detection with Enhanced Token Optimization and Cost Reductioncited by 4
  9. 2025Improving Model Alignment Through Collective Intelligence of Open-Source LLMScited by 4
  10. 2025AttackSeqBench: Benchmarking the Capabilities of LLMs for Attack Sequences Understandingcited by 3
  11. 2026SLMJury: Can Small Language Models Judge as Well as Large Ones?cited by 3
  12. 2025Building Large-Scale English-Romanian Literary Translation Resources with Open Modelscited by 2
  13. 2025From Long to Lean: Performance-aware and Adaptive Chain-of-Thought Compression via Multi-round Refinementcited by 2
  14. 2026Automated Creativity Evaluation of Language Models Across Open-Ended Taskscited by 2
  15. 2025RETuning: Upgrading Inference-Time Scaling for Stock Movement Prediction with Large Language Modelscited by 1
  16. 2026More Bang for the Buck: Improving the Inference of Large Language Models at a Fixed Budget using Reset and Discard (ReD)cited by 1
  17. 2026Reliability-Aware Determinantal Point Processes for Robust Informative Data Selection in Large Language Modelscited by 1
  18. 2026Beyond Accuracy: Decomposing the Reasoning Efficiency of LLMscited by 1
  19. 2026Reasoning as Compression: Unifying Budget Forcing via the Conditional Information Bottleneckcited by 1
  20. 2024FRoG: Evaluating Fuzzy Reasoning of Generalized Quantifiers in Large Language Modelscited by 0
  21. 2024MANTA -- Model Adapter Native generations that's Affordablecited by 0
  22. 2025CLASP: Cost-Optimized LLM-based Agentic System for Phishing Detectioncited by 0
  23. 2026Avoiding Overthinking and Underthinking: Curriculum-Aware Budget Scheduling for LLMscited by 0
  24. 2026When Independent Sampling Outperforms Agentic Reasoningcited by 0
  25. 2026CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compressioncited by 0
  26. 2026Can Agentic Trading Systems Pay for Their Own Intelligence?cited by 0
  27. 2026Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7Bcited by 0
  28. 2026Keep, Customize, or Exit: Default Design and Token Pricing in LLM Reasoning Servicescited by 0
  29. 2026GVS5H: Zero-Shot Self-Orchestration with Ledger-Based Control for Improved LLM Coding Performancecited by 0
  30. 2026LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoningcited by 0
  31. 2026Multi-agent discussion gains less when dissent is withheldcited by 0
  32. 2026How Much Can Language Models Gain from Test-Time Computation?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Reasoning Tax: Token Economics of LLM Reasoning Across Task Types and Deployment Contexts
  2. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs
  3. 2025Plan and Budget: Effective and Efficient Test-Time Scaling on Reasoning Large Language Models
  4. 2026A Comprehensive Evaluation of LLM Reasoning: From Single-Model to Multi-Agent Paradigms
  5. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models
  6. 2025BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokens
  7. 2024Inference Scaling vs Reasoning: An Empirical Analysis of Compute-Optimal LLM Problem-Solving
  8. 2026The Shadow Price of Reasoning: Economic Perspective on Optimal Budget Allocation for LLMs