Data to 7 Oct 2026

Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment

arXiv 2503.21878 ↗First posted 27 Mar 2025cs.AI cs.LG stat.MLcited by 91

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Inference-time computation offers a powerful axis for scaling the performance of language models. However, naively increasing computation in techniques like Best-of-N sampling can lead to performance degradation due to reward hacking. Toward a theoretical understanding of how to best leverage additional computation, we focus on inference-time alignment, which we formalize as the problem of improving the quality of responses drawn from a pre-trained policy, given a prompt of interest and access to an imperfect reward model. We analyze the performance of inference-time alignment algorithms in terms of (i) response quality, and (ii) compute, and provide new results that highlight the importance of the pre-trained policy's coverage over high-quality responses for performance and compute scaling: 1. We show that Best-of-N alignment with an ideal choice for N can achieve optimal performance under stringent notions of coverage, but provably suffers from reward hacking when N is large, and fails to achieve tight guarantees under more realistic coverage conditions. 2. We introduce textttInferenceTimePessimism, a new algorithm which mitigates reward hacking through deliberate use of inference-time compute, implementing the principle of pessimism in the face of uncertainty via rejection sampling; we prove that its performance is optimal and does not degrade with N, meaning it is scaling-monotonic. We complement our theoretical results with an experimental evaluation that demonstrate the benefits of textttInferenceTimePessimism across a variety of tasks and models.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 64 of its 102 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

64 of 102 references.

Closest ancestors · 19

Same direction, newest first

7 more

Other references · 45

2024The Llama 3 Herd of Modelscited by 14,192
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2021Training Verifiers to Solve Math Word Problemscited by 9,743
2020Measuring Massive Multitask Language Understandingcited by 7,768
2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
2024Qwen2.5 Technical Reportcited by 4,713
2023Mistral 7Bcited by 3,065
2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946

33 more

2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
2024Theoretical guarantees on the best-of-n alignment policycited by 116
2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
2024ARC Prize 2024: Technical Reportcited by 103
2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
2023Is RLHF More Difficult than Standard RL?cited by 96
2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
2024How to Evaluate Reward Models for RLHFcited by 84
2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
2024Active Preference Optimization for Sample Efficient RLHFcited by 61
2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
2024The Limits of Inference Scaling Through Resamplingcited by 51
2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
2024Dataset Reset Policy Optimization for RLHFcited by 41
2024West-of-N: Synthetic Preferences for Self-Improving Reward Modelscited by 27
2023Provable Reward-Agnostic Preference-Based Reinforcement Learningcited by 25
2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilizationcited by 25
2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 33; 2026: 58.

91 papers.

Sort:

By area:

2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 46
2025Inference-Time Reward Hacking in Large Language Modelscited by 32
2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 30
2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
2026Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpeningcited by 27
2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 16
2025Let Me Think! A Long Chain-of-Thought Can Be Worth Exponentially Many Short Onescited by 12
2025ROC-n-reroll: How verifier imperfection affects test-time scalingcited by 11
2025Guided Speculative Inference for Efficient Test-Time Alignment of LLMscited by 9
2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
2025PiCSAR: Probabilistic Confidence Selection And Ranking for Reasoning Chainscited by 9
2025Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learningcited by 9

38 more

2025AdaBoN: Adaptive Best-of-N Alignmentcited by 8
2025Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoningcited by 8
2025Best-of-Majority: Minimax-Optimal Strategy for Pass@k Inference Scalingcited by 8
2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
2025Learning to Answer from Correct Demonstrationscited by 6
2025On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inferencecited by 6
2026Likelihood-Based Reward Designs for General LLM Reasoningcited by 6
2025Learning a Pessimistic Reward Model in RLHFcited by 5
2025Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimizationcited by 5
2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hackingcited by 5
2025A*-Decoding: Token-Efficient Inference Scalingcited by 4
2025Making, not Taking, the Best of Ncited by 4
2025Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scalingcited by 4
2026Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlocited by 4
2026On the Generalization Gap in Self-Evolving Language Model Reasoningcited by 4
2025textttSPECS: Faster Test-Time Scaling through Speculative Draftscited by 3
2025Best-of-∞ -- Asymptotic Performance of Test-Time LLM Ensemblingcited by 3
2025Test-time Verification via Optimal Transport: Coverage, ROC, & Sub-optimalitycited by 3
2025ThoughtProbe: Classifier-Guided LLM Thought Space Exploration via Probing Representationscited by 3
2025In Good GRACEs: Principled Teacher Selection for Knowledge Distillationcited by 3
2026ATLAS: Agentic Test-time Learning-to-Allocate Scalingcited by 3
2025SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuningcited by 2
2026Inference-time Unlearning Using Conformal Predictioncited by 2
2026Small Reward Models via Backward Inferencecited by 2
2026PETS: A Principled Framework Towards Optimal Trajectory Allocation for Efficient Test-Time Self-Consistencycited by 2
2026Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignmentcited by 2
2026Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossovercited by 2
2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 2
2026Agentic Systems as Boosting Weak Reasoning Modelscited by 2
2026Understanding Reasoning from Pretraining to Post-Trainingcited by 2
2026TailSFT: Filtered Fine-Tuning Improves Post-Training Performancecited by 2
2026Predicting and improving test-time scaling laws via reward tail-guided searchcited by 1
2026MATA: Multi-Agent Framework for Reliable and Flexible Table Question Answeringcited by 1
2026Temper-Then-Tilt: Principled Unlearning for Generative Models through Tempering and Classifier Guidancecited by 1
2026Power and Limitations of Aggregation in Compound AI Systemscited by 1
2026Provable and Practical In-Context Policy Optimization for Self-Improvementcited by 1
2026Adaptive Generate-Rank-Verify: Inference-Time Search with Costly Verificationcited by 1
2026What Am I Missing? Question-Answering as Hidden State Probingcited by 1

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way