mlsanityData to 7 Oct 2026

How Much Can Language Models Gain from Test-Time Computation?

arXiv 2610.01110 ↗First posted 1 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

How much can test-time computation improve a language model, and at what cost? Test-time scaling is widely proposed as a substitute for larger models, but existing comparisons mostly evaluate one domain at a time and rarely charge selection to the budget. We introduce SELF-POT, a benchmark and evaluation framework that measures the test-time potential of a model across competition mathematics, competitive programming, and agentic workflows. SELF-POT separates candidate coverage from final accuracy on static tasks, tracks correctness transitions under revision, and measures protocol completion alongside task success in agentic environments. Under a unified budget rule, it compares Direct inference with parallel sampling and self-revision under fixed multiples of the Direct budget, and charges every model call, including selection and critique, in dollars. This design supports two kinds of comparison: the gain a model obtains from additional inference, and a lower-cost model with additional inference against a stronger model. Across five low-cost reasoning models on 350 sealed tasks, with Claude Opus 5.5 Direct as the reference, the returns depend on the domain, the selection rule, and failure handling. When we replay the retained programming candidate pools, public-example selection raises correct submissions from 376 to 453 of 500 scheduled cells while saving 12-49% of logical API cost across models, and simply retaining an available candidate when judging fails recovers 61 submissions at unchanged cost. On identical mathematics pools, judging with fallback yields 186 correct submissions versus 182 for voting, while voting saves 12-21% of logical API cost. These controlled replays show how selection and failure handling change the gains realized from the same generated candidates, and they quantify the marginal value of a model judge.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 49 of its 66 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

49 of 66 references.

References · 49

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  8. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  9. 2025s1: Simple test-time scalingcited by 1,311
  10. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  11. 2023Teaching Large Language Models to Self-Debugcited by 963
  12. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
37 more
  1. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  2. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  3. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  4. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
  5. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  6. 2022CodeT: Code Generation with Generated Testscited by 369
  7. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  8. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  9. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  10. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
  11. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  12. 2024Token-Budget-Aware LLM Reasoningcited by 242
  13. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  14. 2024AI Agents That Mattercited by 199
  15. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  16. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155
  17. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  18. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 137
  19. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  20. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  21. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMscited by 111
  22. 2024Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoningcited by 105
  23. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 83
  24. 2024A Unified Approach to Routing and Cascading for LLMscited by 71
  25. 2025Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?cited by 69
  26. 2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computationcited by 67
  27. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 63
  28. 2025Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Modelscited by 52
  29. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  30. 2025Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verificationcited by 49
  31. 2024Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategiescited by 44
  32. 2025Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Aheadcited by 43
  33. 2025When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoningcited by 41
  34. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 39
  35. 2025Cost-of-Pass: An Economic Framework for Evaluating Language Modelscited by 35
  36. 2024Provable Scaling Laws for the Test-Time Compute of Large Language Modelscited by 25
  37. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025The Art of Scaling Test-Time Compute for Large Language Models
  2. 2026Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneck
  3. 2026Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
  4. 2025SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scaling
  5. 2025Scaling Test-time Compute for LLM Agents
  6. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
  7. 2025A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning
  8. 2026Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling