mlsanityData to 7 Oct 2026

Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models

arXiv 2408.00724 ↗First posted 1 Aug 2024cs.AIcited by 282

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

While the scaling laws of large language models (LLMs) training have been extensively studied, optimal inference configurations of LLMs remain underexplored. We study inference scaling laws (aka test-time scaling laws) and compute-optimal inference, focusing on the trade-offs between model sizes and generating additional tokens with different inference strategies. As a first step towards understanding and designing compute-optimal inference methods, we studied cost-performance trade-offs for inference strategies such as greedy search, majority voting, best-of-n, weighted voting, and two different tree search algorithms, using different model sizes and compute budgets. Our findings suggest that scaling inference compute with inference strategies can be more computationally efficient than scaling model parameters. Additionally, smaller models combined with advanced inference algorithms offer Pareto-optimal trade-offs in cost and performance. For example, the Llemma-7B model, when paired with our novel tree search algorithm, consistently outperforms the Llemma-34B model across all tested inference strategies on the MATH benchmark. We hope these insights contribute to a deeper understanding of inference scaling laws (test-time scaling laws) for LLMs.

Key facts. Cited by 282 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 51 of its 60 references that are in the map. First posted on arXiv 1 Aug 2024.

Builds on

51 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing18 Apr
  2. 2023Think before you speak: Training Language Models With Pause Tokens3 Oct
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May
  4. 2023Self-Evaluation Guided Beam Search for Reasoning1 May

Other references · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2021Program Synthesis with Large Language Modelscited by 3,509
  12. 2023Mistral 7Bcited by 3,065
35 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  4. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  7. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  8. 2021Measuring Coding Challenge Competence With APPScited by 879
  9. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  10. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  11. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  12. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  13. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  14. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  15. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  16. 2023Scaling Data-Constrained Language Modelscited by 391
  17. 2023Llemma: An Open Language Model For Mathematicscited by 385
  18. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
  19. 2023The False Promise of Imitating Proprietary LLMscited by 254
  20. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  21. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  22. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelscited by 186
  23. 2023Planning with Large Language Models for Code Generationcited by 179
  24. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 160
  25. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155
  26. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  27. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 129
  28. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoningcited by 96
  29. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  30. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 74
  31. 2021Scaling Scaling Laws with Board Gamescited by 62
  32. 2023Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decodingcited by 59
  33. 2024When is Tree Search Useful for LLM Planning? It Depends on the Discriminatorcited by 46
  34. 2023KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detectioncited by 43
  35. 2024Selecting Large Language Model to Fine-tune via Rectified Scaling Lawcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 167; 2026: 94.

282 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  3. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 225
  4. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  5. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  6. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  7. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 135
  8. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 121
  9. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 94
  10. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 92
  11. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  12. 2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 89
38 more
  1. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  2. 2025Process Reward Models for LLM Agents: Practical Framework and Directionscited by 81
  3. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 80
  4. 2025Efficient Test-Time Scaling via Self-Calibrationcited by 74
  5. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Timecited by 73
  6. 2024ToolGen: Unified Tool Retrieval and Calling via Generationcited by 72
  7. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  8. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 62
  9. 2025Distillation Scaling Lawscited by 59
  10. 2025Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RLcited by 59
  11. 2025Scaling Test-time Compute for LLM Agentscited by 59
  12. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  13. 2024Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewardingcited by 53
  14. 2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?cited by 52
  15. 2025Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Modelscited by 52
  16. 20251.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Trainingcited by 49
  17. 2025Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Searchcited by 48
  18. 2025Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time Thinkingcited by 46
  19. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  20. 2025The Majority is not always right: RL training for solution aggregationcited by 46
  21. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 44
  22. 2025Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Aheadcited by 43
  23. 2025Dynamic Parallel Tree Search for Efficient LLM Reasoningcited by 42
  24. 2025Budget-Aware Tool Use Enables Effective Agent Scalingcited by 42
  25. 2025When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoningcited by 41
  26. 2024Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent Systemcited by 40
  27. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 40
  28. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 38
  29. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  30. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 37
  31. 2025Rethinking External Slow-Thinking: From Snowball Errors to Probability of Correct Reasoningcited by 36
  32. 2025Weight Ensembling Improves Reasoning in Language Modelscited by 36
  33. 2025ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Computecited by 36
  34. 2025Emergent Abilities in Large Language Models: A Surveycited by 34
  35. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  36. 2025METAL: A Multi-Agent Framework for Chart Generation with Test-Time Scalingcited by 33
  37. 2025Retro-Search: Exploring Untaken Paths for Deeper and Efficient Reasoningcited by 32
  38. 2025Scalable Chain of Thoughts via Elastic Reasoningcited by 32

50 most cited of 282 All 282 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025The Art of Scaling Test-Time Compute for Large Language Models
  2. 2025Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signals
  3. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
  4. 2025Are We Scaling the Right Thing? A System Perspective on Test-Time Scaling
  5. 2024A Simple Model of Inference Scaling Laws
  6. 2024Keep Guessing? When Considering Inference Scaling, Mind the Baselines
  7. 2025A*-Decoding: Token-Efficient Inference Scaling
  8. 2025OptScale: Probabilistic Optimality for Inference-time Scaling