mlsanityData to 14 Oct 2026

Evaluation of Best-of-N Sampling Strategies for Language Model Alignment

arXiv 2502.12668 ↗First posted 18 Feb 2025cs.CLcited by 39

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Best-of-N (BoN) sampling with a reward model has been shown to be an effective strategy for aligning Large Language Models (LLMs) with human preferences at the time of decoding. BoN sampling is susceptible to a problem known as reward hacking. Since the reward model is an imperfect proxy for the true objective, an excessive focus on optimizing its value can lead to a compromise of its performance on the true objective. Previous work proposes Regularized BoN sampling (RBoN), a BoN sampling with regularization to the objective, and shows that it outperforms BoN sampling so that it mitigates reward hacking and empirically (Jinnai et al., 2024). However, Jinnai et al. (2024) introduce RBoN based on a heuristic and they lack the analysis of why such regularization strategy improves the performance of BoN sampling. The aim of this study is to analyze the effect of BoN sampling on regularization strategies. Using the regularization strategies corresponds to robust optimization, which maximizes the worst case over a set of possible perturbations in the proxy reward. Although the theoretical guarantees are not directly applicable to RBoN, RBoN corresponds to a practical implementation. This paper proposes an extension of the RBoN framework, called Stochastic RBoN sampling (SRBoN), which is a theoretically guaranteed approach to worst-case RBoN in proxy reward. We then perform an empirical evaluation using the AlpacaFarm and Anthropic's hh-rlhf datasets to evaluate which factors of the regularization strategies contribute to the improvement of the true proxy reward. In addition, we also propose another simple RBoN method, the Sentence Length Regularized BoN, which has a better performance in the experiment as compared to the previous methods.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 69 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

50 of 69 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization10 Jul
  2. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback13 Jun
  3. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling2 Jun
  4. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  5. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  6. 2024ARGS: Alignment as Reward-Guided Search23 Jan
  7. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  8. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 41

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023Mistral 7Bcited by 3,164
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2021Convex optimizationcited by 2,515
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
29 more
  1. 2016Concrete Problems in AI Safetycited by 1,642
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  3. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  7. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  8. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  9. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  10. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  11. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  12. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  13. 2019A Theory of Regularized Markov Decision Processescited by 286
  14. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  15. 2017A unified view of entropy-regularized Markov decision processescited by 182
  16. 2024More Agents Is All You Needcited by 180
  17. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  18. 2023Controlled Decoding from Language Modelscited by 135
  19. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  20. 2021Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translationcited by 65
  21. 2023LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Modelscited by 60
  22. 2022Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decodingcited by 54
  23. 2023It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Riskcited by 51
  24. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 47
  25. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  26. 2021Regularized Policies are Reward Robustcited by 32
  27. 2022Your Policy Regularizer is Secretly an Adversarycited by 21
  28. 2024Improving Minimum Bayes Risk Decoding with Multi-Promptcited by 13
  29. 2014An Adversarial Interpretation of Information-Theoretic Bounded Rationalitycited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 21.

39 papers.

Sort:

By area:

  1. 2025MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinementcited by 58
  2. 2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 46
  3. 2025Inference-Time Reward Hacking in Large Language Modelscited by 34
  4. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  5. 2025Soft Best-of-n Sampling for Model Alignmentcited by 30
  6. 2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustnesscited by 20
  7. 2025Large Reasoning Models Learn Better Alignment from Flawed Thinkingcited by 13
  8. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
  9. 2025Solve-Detect-Verify: Inference-Time Scaling with Flexible Generative Verifiercited by 8
  10. 2025DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesiscited by 8
  11. 2025Majority of the Bests: Improving Best-of-N via Bootstrappingcited by 7
  12. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
27 more
  1. 2025Making, not Taking, the Best of Ncited by 6
  2. 2025Semantic Probabilistic Control of Language Modelscited by 4
  3. 2026ReThinker: Scientific Reasoning by Rethinking with Guided Reflection and Confidence Controlcited by 4
  4. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  5. 2026FUSE: Ensembling Verifiers with Zero Labeled Datacited by 4
  6. 2026AVIS: Adaptive Test-Time Scaling for Vision-Language Modelscited by 4
  7. 2025Optimal Aggregation of LLM and PRM Signals for Efficient Test-Time Scalingcited by 3
  8. 2025The Reward Model Selection Crisis in Personalized Alignmentcited by 3
  9. 20263D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligencecited by 3
  10. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 3
  11. 2026Flexible Empowerment at Reasoning with Extended Best-of-N Samplingcited by 3
  12. 2025Calibrating Translation Decoding with Quality Estimation on LLMscited by 2
  13. 2025Video Parallel Scaling: Aggregating Diverse Frame Subsets for VideoLLMscited by 2
  14. 2025Re-evaluating Minimum Bayes Risk Decoding for Automatic Speech Recognitioncited by 2
  15. 2026Inference-time Unlearning Using Conformal Predictioncited by 2
  16. 2025Adaptive Blockwise Search: Inference-Time Alignment for Large Language Modelscited by 1
  17. 2025Test-time reward-guided alignment of language models by importance sampling on pre-logit spacecited by 0
  18. 2026DARC: Disagreement-Aware Alignment via Risk-Constrained Decodingcited by 0
  19. 2026Justified or Just Convincing? Error Verifiability as a Dimension of LLM Qualitycited by 0
  20. 2026LACE: Lattice Attention for Cross-thread Explorationcited by 0
  21. 2026From 'May' to 'Is': Certainty Distortion in Language Model Rewritingcited by 0
  22. 2026Gradient-Guided Reward Optimization for Inference-time Alignmentcited by 0
  23. 2026Mitigating Over-Optimization in PRM-Guided Search in Mathematical Reasoning by Optimizing the Guidecited by 0
  24. 2026ExpBoN: Exponential-Noise Best-of-n for Efficient Test-Time LLM Alignmentcited by 0
  25. 2026AdaPilot: Towards Scene-Adaptive Policy Learning for Cross-Generator Text-to-Image Quality Optimizationcited by 0
  26. 2026Provable Test-Time Scaling for Beam Search in LLM Reasoningcited by 0
  27. 2026Best-of-N Guidance for Test-time Diffusion Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment
  2. 2024Variational Best-of-N Alignment
  3. 2026Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignment
  4. 2025AdaBoN: Adaptive Best-of-N Alignment
  5. 2024Bayesian Reward Models for LLM Alignment
  6. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decoding
  7. 2024Value Augmented Sampling for Language Model Alignment and Personalization
  8. 2024A Probability--Quality Trade-off in Aligned Language Models and its Relation to Sampling Adaptors