mlsanityData to 14 Oct 2026

Soft Best-of-n Sampling for Model Alignment

arXiv 2505.03156 ↗First posted 6 May 2025cs.IT cs.AI math.ITcited by 30

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Best-of-n (BoN) sampling is a practical approach for aligning language model outputs with human preferences without expensive fine-tuning. BoN sampling is performed by generating n responses to a prompt and then selecting the sample that maximizes a reward function. BoN yields high reward values in practice at a distortion cost, as measured by the KL-divergence between the sampled and original distribution. This distortion is coarsely controlled by varying the number of samples: larger n yields a higher reward at a higher distortion cost. We introduce Soft Best-of-n sampling, a generalization of BoN that allows for smooth interpolation between the original distribution and reward-maximizing distribution through a temperature parameter λ. We establish theoretical guarantees showing that Soft Best-of-n sampling converges sharply to the optimal tilted distribution at a rate of O(1/n) in KL and the expected (relative) reward. For sequences of discrete outputs, we analyze an additive reward model that reveals the fundamental limitations of blockwise sampling.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 46 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

21 of 46 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignment18 Feb
  2. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling18 Oct
  3. 2024Variational Best-of-N Alignment8 Jul
  4. 2024Information Theoretic Guarantees For Policy Alignment In Large Language Models9 Jun
  5. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling2 Jun
  6. 2024Asymptotics of Language Model Alignment2 Apr
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 14

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  11. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  12. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
2 more
  1. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  2. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 76

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 21.

30 papers.

Sort:

By area:

  1. 2025Inference-Time Reward Hacking in Large Language Modelscited by 34
  2. 2025ImAgent: A Unified Multimodal Agent Framework for Test-Time Scalable Image Generationcited by 17
  3. 2025Guided Speculative Inference for Efficient Test-Time Alignment of LLMscited by 11
  4. 2025Best-of-Majority: Minimax-Optimal Strategy for Pass@k Inference Scalingcited by 9
  5. 2026VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Modelcited by 9
  6. 2026VIGOR: VIdeo Geometry-Oriented Reward for Temporal Generative Alignmentcited by 7
  7. 2026Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibilitycited by 6
  8. 2025Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectoriescited by 4
  9. 2025textttSPECS: Faster Test-Time Scaling through Speculative Draftscited by 3
  10. 2026Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignmentcited by 3
  11. 2026Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignmentcited by 3
  12. 2026Flexible Empowerment at Reasoning with Extended Best-of-N Samplingcited by 3
18 more
  1. 2026DPRM: A Plug-in Doob h transform-induced Token-Ordering Module for Discrete Diffusion Modelscited by 3
  2. 2026Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learningcited by 2
  3. 2025Connections between reinforcement learning with feedback,test-time scaling, and diffusion guidance: An anthologycited by 1
  4. 2025STARS: Synchronous Token Alignment for Robust Supervision in Large Language Modelscited by 1
  5. 2026Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPOcited by 1
  6. 2026Reward Learning from Best-of-N Preference Data: Targets, Tradeoffs, and Design Principlescited by 1
  7. 2026Granularity-Regulated Adaptive Computational Efficiency for Optimal Verification in Test-Time Scalingcited by 1
  8. 2025RoBoN: Routed Online Best-of-n for Test-Time Scaling with Multiple LLMscited by 0
  9. 2026Multimodal Diffusion to Mutually Enhance Polarized Light and Low Resolution EBSD Datacited by 0
  10. 2026Theoretical Limits of Language Model Alignmentcited by 0
  11. 2026Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignmentcited by 0
  12. 2026Best-of-Better-N: Generating Pre-Aligned Responses with In-Context Learningcited by 0
  13. 2026Privacy Without Regret: Differentially Private Inference-Time Alignmentcited by 0
  14. 2026ExpBoN: Exponential-Noise Best-of-n for Efficient Test-Time LLM Alignmentcited by 0
  15. 2026Uniform Race: Parameter-Free Approximate Rejection Samplingcited by 0
  16. 2026Revisiting scaling laws for reward optimizationcited by 0
  17. 2026Holdout Best-of-N: Unbiased Evaluation and Its Costcited by 0
  18. 2026Universal Bounds for Self-Normalized Kullback-Leibler Estimationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment
  2. 2025AdaBoN: Adaptive Best-of-N Alignment
  3. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment
  4. 2025Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis
  5. 2024A Probability--Quality Trade-off in Aligned Language Models and its Relation to Sampling Adaptors
  6. 2025Majority of the Bests: Improving Best-of-N via Bootstrapping
  7. 2026Why Self-Rewarding Works: Theoretical Guarantees for Iterative Alignment of Language Models
  8. 2026Tail Competition Explains Scaling in Best-of-N Sampling for Verifiable Problems