mlsanityData to 14 Oct 2026

BOND: Aligning LLMs with Best-of-N Distillation

arXiv 2407.14622 ↗First posted 19 Jul 2024cs.LG cs.AI cs.CLcited by 76

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) is a key driver of quality and safety in state-of-the-art large language models. Yet, a surprisingly simple and strong inference-time strategy is Best-of-N sampling that selects the best generation among N candidates. In this paper, we propose Best-of-N Distillation (BOND), a novel RLHF algorithm that seeks to emulate Best-of-N but without its significant computational overhead at inference time. Specifically, BOND is a distribution matching algorithm that forces the distribution of generations from the policy to get closer to the Best-of-N distribution. We use the Jeffreys divergence (a linear combination of forward and backward KL) to balance between mode-covering and mode-seeking behavior, and derive an iterative formulation that utilizes a moving anchor for efficiency. We demonstrate the effectiveness of our approach and several design choices through experiments on abstractive summarization and Gemma models. Aligning Gemma policies with BOND outperforms other RLHF algorithms by improving results on several benchmarks.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 70 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

48 of 70 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024WARP: On the Benefits of Weight Averaged Rewarded Policies24 Jun
  2. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  3. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  4. 2023Nash Learning from Human Feedback1 Dec
  5. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  9. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  10. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  11. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  12. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2023Mistral 7Bcited by 3,164
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
24 more
  1. 2020Learning to summarize from human feedbackcited by 2,588
  2. 2024Mixtral of Expertscited by 1,823
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  6. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  8. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  9. 2019A Theory of Regularized Markov Decision Processescited by 286
  10. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  11. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  12. 2024CodeGemma: Open Code Models Based on Gemmacited by 150
  13. 2018Supervising strong learners by amplifying weak expertscited by 145
  14. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  15. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  16. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  17. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  18. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 72
  19. 2020Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learningcited by 66
  20. 2024Learn Your Reference Model for Real Good Alignmentcited by 57
  21. 2024Variational Best-of-N Alignmentcited by 51
  22. 2024Asymptotics of Language Model Alignmentcited by 47
  23. 2023Tuning computer vision models with task rewardscited by 37
  24. 2024Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashioncited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 34; 2026: 23.

76 papers.

Sort:

By area:

  1. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  2. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  3. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
  4. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  5. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  6. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
  7. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 76
  8. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 72
  9. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 71
  10. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 70
  11. 2025Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?cited by 70
  12. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 63
38 more
  1. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 61
  2. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  3. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoningcited by 52
  4. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 50
  5. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 49
  6. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  7. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 42
  8. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 42
  9. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  10. 2025Weight Ensembling Improves Reasoning in Language Modelscited by 38
  11. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  12. 2024InfAlign: Inference-aware language model alignmentcited by 35
  13. 2025Inference-Time Reward Hacking in Large Language Modelscited by 34
  14. 2025Soft Best-of-n Sampling for Model Alignmentcited by 30
  15. 2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedbackcited by 26
  16. 2025AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidencecited by 24
  17. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 22
  18. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignmentcited by 20
  19. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  20. 2025Learning from Peers in Reasoning Modelscited by 16
  21. 2025PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Spacecited by 16
  22. 2025The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisationcited by 16
  23. 2025Sample Complexity and Representation Ability of Test-time Scaling Paradigmscited by 12
  24. 2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Gamecited by 10
  25. 2024Diversity-Rewarded CFG Distillationcited by 9
  26. 2024Faster WIND: Accelerating Iterative Best-of-N Distillation for LLM Alignmentcited by 9
  27. 2026VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Modelcited by 9
  28. 2026Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-trainingcited by 8
  29. 2026Vector Policy Optimization: Training for Diversity Improves Test-Time Searchcited by 8
  30. 2025LLM Alignment as Retriever Optimization: An Information Retrieval Perspectivecited by 7
  31. 2025Bias Fitting to Mitigate Length Bias of Reward Model in RLHFcited by 7
  32. 2025Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scalingcited by 7
  33. 2026LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Modelscited by 7
  34. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gapcited by 6
  35. 2025Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functionscited by 6
  36. 2025On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inferencecited by 6
  37. 2025Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generatorcited by 5
  38. 2026Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillationcited by 5

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  3. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  4. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  6. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
  7. 2025Greedy Sampling Is Provably Efficient for RLHF
  8. 2024Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data