# Soft Best-of-n Sampling for Model Alignment

arXiv: https://arxiv.org/abs/2505.03156 · HTML: https://mlsanity.com/paper/2505.03156/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 21 of its 46 references that are in the map.
- First posted on arXiv 6 May 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Theoretical guarantees on the best-of-n alignment policy](https://mlsanity.com/paper/2401.01879/) (2024)
- [BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling](https://mlsanity.com/paper/2406.00832/) (2024)
- [BOND: Aligning LLMs with Best-of-N Distillation](https://mlsanity.com/paper/2407.14622/) (2024)
- [TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling](https://mlsanity.com/paper/2410.16033/) (2024)
- [Variational Best-of-N Alignment](https://mlsanity.com/paper/2407.06057/) (2024)
- [Asymptotics of Language Model Alignment](https://mlsanity.com/paper/2404.01730/) (2024)
- [Evaluation of Best-of-N Sampling Strategies for Language Model Alignment](https://mlsanity.com/paper/2502.12668/) (2025)
- [Information Theoretic Guarantees For Policy Alignment In Large Language Models](https://mlsanity.com/paper/2406.05883/) (2024)

## Built on by (most cited first, 30 of 30)

- [Inference-Time Reward Hacking in Large Language Models](https://mlsanity.com/paper/2506.19248/) (2025)
- [ImAgent: A Unified Multimodal Agent Framework for Test-Time Scalable Image Generation](https://mlsanity.com/paper/2511.11483/) (2025)
- [Guided Speculative Inference for Efficient Test-Time Alignment of LLMs](https://mlsanity.com/paper/2506.04118/) (2025)
- [Best-of-Majority: Minimax-Optimal Strategy for Pass@k Inference Scaling](https://mlsanity.com/paper/2510.03199/) (2025)
- [VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Model](https://mlsanity.com/paper/2605.01194/) (2026)
- [VIGOR: VIdeo Geometry-Oriented Reward for Temporal Generative Alignment](https://mlsanity.com/paper/2603.16271/) (2026)
- [Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility](https://mlsanity.com/paper/2608.04001/) (2026)
- [Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectories](https://mlsanity.com/paper/2511.07368/) (2025)
- [textttSPECS: Faster Test-Time Scaling through Speculative Drafts](https://mlsanity.com/paper/2506.15733/) (2025)
- [Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignment](https://mlsanity.com/paper/2603.05739/) (2026)
- [Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignment](https://mlsanity.com/paper/2603.06797/) (2026)
- [Flexible Empowerment at Reasoning with Extended Best-of-N Sampling](https://mlsanity.com/paper/2604.15614/) (2026)
- [DPRM: A Plug-in Doob h transform-induced Token-Ordering Module for Discrete Diffusion Models](https://mlsanity.com/paper/2604.24357/) (2026)
- [Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learning](https://mlsanity.com/paper/2606.03962/) (2026)
- [Connections between reinforcement learning with feedback,test-time scaling, and diffusion guidance: An anthology](https://mlsanity.com/paper/2509.04372/) (2025)
- [STARS: Synchronous Token Alignment for Robust Supervision in Large Language Models](https://mlsanity.com/paper/2511.03827/) (2025)
- [Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO](https://mlsanity.com/paper/2602.01603/) (2026)
- [Reward Learning from Best-of-N Preference Data: Targets, Tradeoffs, and Design Principles](https://mlsanity.com/paper/2605.30619/) (2026)
- [Granularity-Regulated Adaptive Computational Efficiency for Optimal Verification in Test-Time Scaling](https://mlsanity.com/paper/2606.19354/) (2026)
- [RoBoN: Routed Online Best-of-n for Test-Time Scaling with Multiple LLMs](https://mlsanity.com/paper/2512.05542/) (2025)
- [Multimodal Diffusion to Mutually Enhance Polarized Light and Low Resolution EBSD Data](https://mlsanity.com/paper/2604.22212/) (2026)
- [Theoretical Limits of Language Model Alignment](https://mlsanity.com/paper/2605.07105/) (2026)
- [Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignment](https://mlsanity.com/paper/2605.13537/) (2026)
- [Best-of-Better-N: Generating Pre-Aligned Responses with In-Context Learning](https://mlsanity.com/paper/2607.03453/) (2026)
- [Privacy Without Regret: Differentially Private Inference-Time Alignment](https://mlsanity.com/paper/2608.26324/) (2026)
- [ExpBoN: Exponential-Noise Best-of-n for Efficient Test-Time LLM Alignment](https://mlsanity.com/paper/2609.21899/) (2026)
- [Uniform Race: Parameter-Free Approximate Rejection Sampling](https://mlsanity.com/paper/2609.34639/) (2026)
- [Revisiting scaling laws for reward optimization](https://mlsanity.com/paper/2609.38526/) (2026)
- [Holdout Best-of-N: Unbiased Evaluation and Its Cost](https://mlsanity.com/paper/2610.08719/) (2026)
- [Universal Bounds for Self-Normalized Kullback-Leibler Estimation](https://mlsanity.com/paper/2610.12851/) (2026)

## Abstract (arXiv)

Best-of-n (BoN) sampling is a practical approach for aligning language model outputs with human preferences without expensive fine-tuning. BoN sampling is performed by generating n responses to a prompt and then selecting the sample that maximizes a reward function. BoN yields high reward values in practice at a distortion cost, as measured by the KL-divergence between the sampled and original distribution. This distortion is coarsely controlled by varying the number of samples: larger n yields a higher reward at a higher distortion cost. We introduce Soft Best-of-n sampling, a generalization of BoN that allows for smooth interpolation between the original distribution and reward-maximizing distribution through a temperature parameter λ. We establish theoretical guarantees showing that Soft Best-of-n sampling converges sharply to the optimal tilted distribution at a rate of O(1/n) in KL and the expected (relative) reward. For sequences of discrete outputs, we analyze an additive reward model that reveals the fundamental limitations of blockwise sampling.