mlsanityData to 14 Oct 2026

What should post-training optimize? A test-time scaling law perspective

arXiv 2605.10716 ↗First posted 11 May 2026cs.LG stat.MLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Large language models are increasingly deployed with test-time strategies: sample N responses, score them with a reward model or verifier, and return the best. This deployment rule exposes a mismatch in post-training: standard objectives optimize the mean reward of a single response, whereas best-of-N performance is governed by the upper tail of the reward distribution. Recent test-time-aware objectives partly address this mismatch, but typically assume that training can use the same per-prompt rollout budget as deployment, which is impractical when post-training must cover many prompts while deployment can allocate much larger per-prompt test-time compute. We study this budget-mismatch regime, where only mll N per-prompt rollouts are available during training but the target objective is best-of-N deployment. Under structural assumptions on the reward tails, we show that the policy gradient of the best-of-N objective can be approximated from a much smaller rollout group by extrapolating upper-tail statistics. This yields a family of Tail-Extrapolated estimators for best-of-N-oriented post-training: a simple direct estimator, Tail-Extrapolated Advantage (TEA), and a fixed-order debiased Prefix-TEA estimator based on moment cancellation. Experiments on instruction-following tasks show that TEA and Prefix-TEA improve best-of-N performance across different language models, reward models and datasets under various training and test-time budget settings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 26 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

19 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay5 Jun

Other references · 18

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  6. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  7. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 867
  8. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  9. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  10. 2026Learning to Discover at Test Timecited by 115
  11. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  12. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
6 more
  1. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 73
  2. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 53
  3. 2024InfAlign: Inference-aware language model alignmentcited by 35
  4. 2025The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisationcited by 16
  5. 2026Predicting and improving test-time scaling laws via reward tail-guided searchcited by 2
  6. 2026Compute Aligned Training: Optimizing for Test Time Inferencecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training
  2. 2026Test-Time Scaling Makes Overtraining Compute-Optimal
  3. 2026Consolidating Rewarded Perturbations for LLM Post-Training
  4. 2026Sharpening Tax in Post-Training
  5. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling
  6. 2025Towards a Unified View of Large Language Model Post-Training
  7. 2026Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Training
  8. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training