# Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility

arXiv: https://arxiv.org/abs/2608.04001 · HTML: https://mlsanity.com/paper/2608.04001/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 31 of its 152 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 30 of 31)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models](https://mlsanity.com/paper/2408.00724/) (2024)
- [Universal Self-Consistency for Large Language Model Generation](https://mlsanity.com/paper/2311.17311/) (2023)
- [AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset](https://mlsanity.com/paper/2504.16891/) (2025)
- [Phi-4-reasoning Technical Report](https://mlsanity.com/paper/2504.21318/) (2025)
- [Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations](https://mlsanity.com/paper/2411.00640/) (2024)
- [LIMR: Less is More for RL Scaling](https://mlsanity.com/paper/2502.11886/) (2025)
- [Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs](https://mlsanity.com/paper/2605.00674/) (2026)
- [OpenCodeReasoning: Advancing Data Distillation for Competitive Coding](https://mlsanity.com/paper/2504.01943/) (2025)
- [NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model](https://mlsanity.com/paper/2508.14444/) (2025)
- [AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy](https://mlsanity.com/paper/2506.13284/) (2025)
- [It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Risk](https://mlsanity.com/paper/2310.01387/) (2023)
- [Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators](https://mlsanity.com/paper/2504.15253/) (2025)
- [Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math](https://mlsanity.com/paper/2504.21233/) (2025)
- [Faster Minimum Bayes Risk Decoding with Confidence-based Pruning](https://mlsanity.com/paper/2311.14919/) (2023)
- [Inference-Time Reward Hacking in Large Language Models](https://mlsanity.com/paper/2506.19248/) (2025)
- [EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes](https://mlsanity.com/paper/2507.11407/) (2025)
- [Soft Best-of-n Sampling for Model Alignment](https://mlsanity.com/paper/2505.03156/) (2025)
- [LLM-as-a-Verifier: A General-Purpose Verification Framework](https://mlsanity.com/paper/2607.05391/) (2026)
- [Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores](https://mlsanity.com/paper/2410.03492/) (2024)
- [OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critique](https://mlsanity.com/paper/2507.09075/) (2025)
- [Hyperparameter-Free Approach for Faster Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2401.02749/) (2024)
- [We need to talk about random seeds](https://mlsanity.com/paper/2210.13393/) (2022)
- [TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillation](https://mlsanity.com/paper/2503.04872/) (2025)
- [FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion](https://mlsanity.com/paper/2503.04222/) (2025)

## Built on by (most cited first, 6 of 6)

- [GRASP: Reinforcing Language Model Anonymizers with Group Relative Policy Optimization](https://mlsanity.com/paper/2608.06526/) (2026)
- [Faults That Fortify: CNN Adversarial Robustness via GPU Undervolting](https://mlsanity.com/paper/2608.20572/) (2026)
- [Recursive Agentic Reasoning](https://mlsanity.com/paper/2608.23956/) (2026)
- [Budget Boundary Effects in Test-Time Mathematical Reasoning](https://mlsanity.com/paper/2609.38699/) (2026)
- [Adaptive Power Sampling for LLM Reasoning](https://mlsanity.com/paper/2610.08563/) (2026)
- [Collaborative Learning and Reasoning over Model Slices](https://mlsanity.com/paper/2610.12927/) (2026)

## Abstract (arXiv)

Large language models can solve harder reasoning problems with more inference-time compute. The term "test-time scaling," however, covers several inference algorithms: extending deliberation along one trajectory, sampling completed candidates and aggregating them by voting or verification, and searching over partial states. These algorithms differ in statistical structure, compute requirements, and failure modes. Treating them as interchangeable under a scalar "budget," or reporting accuracy without specifying the inference protocol, makes results difficult to compare across studies. We study test-time scaling along three axes. First, we formalize it as budgeted inference over the implicit prefix tree of an autoregressive model and distinguish single-trajectory sequential scaling, leaf-level scaling with terminal reduction, and prefix-level scaling. Second, we treat the full inference system as the evaluated object and separate end-to-end performance from candidate-bank diagnostics. We introduce an evaluation profile whose coordinates and simple functionals recover or bound common repeated-sampling metrics, and require compute accounting and uncertainty estimates that match the protocol. Third, we distinguish exact replay from distributional reproducibility and state the requirements for each. We also organize open-weight reasoning models by model-side and interface mechanisms. Our empirical study covers broad knowledge, symbolic reasoning, and competition mathematics, and we publicly release 1,403,520 sampled model attempts. The project website is available at https://mohsenhariri.github.io/scorio/tts. The released datasets are Trace (https://huggingface.co/datasets/harimo/scorio-trace), Lite (https://huggingface.co/datasets/harimo/scorio-lite), Math (https://huggingface.co/buckets/harimo/scorio-math), and SuperGPQA (https://huggingface.co/buckets/harimo/scorio-gpqa).