mlsanityData to 7 Oct 2026

Ranking Reasoning LLMs under Test-Time Scaling

arXiv 2603.10960 ↗First posted 11 Mar 2026cs.LG math.ST stat.THcited by 1

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Test-time scaling evaluates reasoning LLMs by sampling multiple outputs per prompt, but ranking models in this regime remains underexplored. We formalize dense benchmark ranking under test-time scaling and introduce Scorio, a library that implements statistical ranking methods such as paired-comparison models, item response theory (IRT) models, voting rules, and graph- and spectral-based methods. Across 20 reasoning models on four Olympiad-style math benchmarks (AIME'24, AIME'25, HMMT'25, and BrUMO'25; up to N=80 trials), most full-trial rankings agree closely with the Bayesian gold standard BayesU@80 (mean Kendall's τb = 0.93--0.95), and 19--34 methods recover exactly the same ordering. In the single-trial regime, the best methods reach τb ≈ 0.86. Using greedy decoding as an empirical prior (BayesR0@N) reduces variance at N=1 by 16--52%, but can bias rankings when greedy and stochastic sampling disagree. These results identify reliable ranking methods for both high- and low-budget test-time scaling. We release Scorio as an open-source library at https://github.com/mohsenhariri/scorio.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 19 of its 81 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

19 of 81 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model20 Aug
  2. 2025gpt-oss-120b & gpt-oss-20b Model Card8 Aug
  3. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug

Other references · 16

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  7. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  8. 2025LIMO: Less is More for Reasoningcited by 488
  9. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  10. 2025Phi-4-reasoning Technical Reportcited by 146
  11. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergycited by 75
  12. 2025Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theorycited by 42
4 more
  1. 2025EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modescited by 26
  2. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 23
  3. 2024A Statistical Framework for Ranking LLM-Based Chatbotscited by 11
  4. 2019Davidson-Luce model for multi-item choice with tiescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Scorio.jl: A Julia package for ranking stochastic responsescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
  2. 2026Overwhelmed by Choice: Studying LLM Decision Making at Scale
  3. 2025ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning
  4. 2025Mitigating Strategy-Selection Bias in Reasoning for More Effective Test-Time Scaling
  5. 2026Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning
  6. 2025Rethinking the Role of Prompting Strategies in LLM Test-Time Scaling: A Perspective of Probability Theory
  7. 2025Scaling over Scaling: Exploring Test-Time Scaling Plateau in Large Reasoning Models
  8. 2026Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals