mlsanityData to 7 Oct 2026

BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling

arXiv 2406.00832 ↗First posted 2 Jun 2024cs.CL cs.LGcited by 120

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

This paper concerns the problem of aligning samples from large language models to human preferences using best-of-n sampling, where we draw n samples, rank them, and return the best one. We consider two fundamental problems. First: what is the relationship between best-of-n and approaches to alignment that train LLMs to output samples with a high expected reward (e.g., RLHF or DPO)? To answer this, we embed both the best-of-n distribution and the sampling distributions learned by alignment procedures in a common class of tiltings of the base LLM distribution. We then show that, within this class, best-of-n is essentially optimal in terms of the trade-off between win-rate against the base model vs KL distance from the base model. That is, best-of-n is the best choice of alignment distribution if the goal is to maximize win rate. However, best-of-n requires drawing n samples for each inference, a substantial cost. To avoid this, the second problem we consider is how to fine-tune a LLM to mimic the best-of-n sampling distribution. We derive BoNBoN Alignment to achieve this by exploiting the special structure of the best-of-n distribution. Experiments show that BoNBoN alignment yields substantial improvements in producing a model that is preferred to the base policy while minimally affecting off-target aspects.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 43 references that are in the map. First posted on arXiv 2 Jun 2024.

Builds on

33 of 43 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  2. 2024Asymptotics of Language Model Alignment2 Apr
  3. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  4. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  6. 2023Large Language Model Alignment: A Survey26 Sep
  7. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  12. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
13 more
  1. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  2. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  3. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  4. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  6. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  7. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  8. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  9. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  10. 2023Controlled Decoding from Language Modelscited by 133
  11. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  12. 2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
  13. 2024Transforming and Combining Rewards for Aligning Large Language Modelscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 65; 2026: 36.

120 papers.

Sort:

By area:

  1. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  2. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  3. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  4. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  5. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  6. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
  7. 2025Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?cited by 69
  8. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  9. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  10. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 63
  11. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 62
  12. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
38 more
  1. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60
  2. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  3. 2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?cited by 52
  4. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoningcited by 51
  5. 2024Variational Best-of-N Alignmentcited by 49
  6. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  7. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 49
  8. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  9. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 46
  10. 2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 45
  11. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  12. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  13. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 38
  14. 2025A Survey of Direct Preference Optimizationcited by 35
  15. 2024InfAlign: Inference-aware language model alignmentcited by 34
  16. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  17. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  18. 2025Soft Best-of-n Sampling for Model Alignmentcited by 28
  19. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  20. 2025Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decodingcited by 26
  21. 2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedbackcited by 26
  22. 2024Self-rationalization improves LLM as a fine-grained judgecited by 25
  23. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forwardcited by 24
  24. 2025Sample, Don't Search: Rethinking Test-Time Alignment for Language Modelscited by 21
  25. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 20
  26. 2026How RLHF Amplifies Sycophancycited by 20
  27. 2026Reward Hacking in Rubric-Based Reinforcement Learningcited by 18
  28. 2024Training Language Models to Win Debates with Self-Play Improves Judge Accuracycited by 16
  29. 2025Learning from Peers in Reasoning Modelscited by 16
  30. 2025AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Taskscited by 16
  31. 2025The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisationcited by 16
  32. 2025Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Modelscited by 15
  33. 2025CoDe: Blockwise Control for Denoising Diffusion Modelscited by 14
  34. 2025PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Spacecited by 14
  35. 2025MARS-SQL: A multi-agent reinforcement learning framework for Text-to-SQLcited by 14
  36. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 12
  37. 2025Personalized LLM Decoding via Contrasting Personal Preferencecited by 12
  38. 2025Test-time Prompt Interventioncited by 12

50 most cited of 120 All 120 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment
  2. 2026Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignment
  3. 2025Majority of the Bests: Improving Best-of-N via Bootstrapping
  4. 2024Sample-Efficient Alignment for LLMs
  5. 2024A Probability--Quality Trade-off in Aligned Language Models and its Relation to Sampling Adaptors
  6. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics
  7. 2025ComPO: Preference Alignment via Comparison Oracles
  8. 2024Efficient Alignment of Large Language Models via Data Sampling