mlsanityData to 7 Oct 2026

Scaling LLM Inference with Optimized Sample Compute Allocation

arXiv 2410.22480 ↗First posted 29 Oct 2024cs.CL cs.AIcited by 16

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Sampling is a basic operation in many inference-time algorithms of large language models (LLMs). To scale up inference efficiently with a limited compute, it is crucial to find an optimal allocation for sample compute budgets: Which sampling configurations (model, temperature, language, etc.) do we use? How many samples do we generate in each configuration? We formulate these choices as a learning problem and propose OSCA, an algorithm that Optimizes Sample Compute Allocation by finding an optimal mix of different inference configurations. Our experiments show that with our learned mixed allocation, we can achieve accuracy better than the best single configuration with 128x less compute on code generation and 25x less compute on 4 reasoning tasks. OSCA is also shown to be effective in agentic workflows beyond single-turn tasks, achieving a better accuracy on SWE-Bench with 3x less compute than the default configuration. Our code and generations are released at https://github.com/LeiLiLab/OSCA.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 23 of its 30 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

23 of 30 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computation7 Oct
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  3. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing18 Apr
  4. 2024THOUGHTSCULPT: Reasoning with Intermediate Revision and Search9 Apr
  5. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar

Other references · 18

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  7. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  8. 2018Universal Transformerscited by 613
  9. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  10. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  11. 2022CodeT: Code Generation with Generated Testscited by 369
  12. 2022HyperTree Proof Search for Neural Theorem Provingcited by 192
6 more
  1. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  2. 2022Language Model Cascadescited by 104
  3. 2023Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistencycited by 46
  4. 2023ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifierscited by 41
  5. 2023Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inferencecited by 40
  6. 2024LLM can Achieve Self-Regulation via Hyperparameter Aware Generationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 1.

16 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  3. 2025SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scalingcited by 54
  4. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  5. 2025Optimizing Temperature for Language Models with Multi-Sample Inferencecited by 33
  6. 2025METAL: A Multi-Agent Framework for Chart Generation with Test-Time Scalingcited by 33
  7. 2024Provable Scaling Laws for the Test-Time Compute of Large Language Modelscited by 25
  8. 2024Syzygy: Dual Code-Test C to (safe) Rust Translation using LLMs and Dynamic Analysiscited by 16
  9. 2025ETS: Efficient Tree Search for Inference-Time Scalingcited by 15
  10. 2025From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuningcited by 12
  11. 2025Rethinking the Role of Prompting Strategies in LLM Test-Time Scaling: A Perspective of Probability Theorycited by 10
  12. 2025AdaBoN: Adaptive Best-of-N Alignmentcited by 8
4 more
  1. 2025Latency and Token-Aware Test-Time Computecited by 8
  2. 2025DynScaling: Efficient Verifier-free Inference Scaling via Dynamic and Integrated Samplingcited by 7
  3. 2026On Calibration of Large Language Models: From Response To Capabilitycited by 5
  4. 2025Plan-and-Refine: Diverse and Comprehensive Retrieval-Augmented Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMs
  2. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models
  3. 2025OptScale: Probabilistic Optimality for Inference-time Scaling
  4. 2025Generalizing Test-time Compute-optimal Scaling as an Optimizable Graph
  5. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents
  6. 2026Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization
  7. 2026InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents
  8. 2025Scaling Test-time Compute for LLM Agents