mlsanityData to 7 Oct 2026

Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment

arXiv 2404.01054 ↗First posted 1 Apr 2024cs.CL cs.AIcited by 24

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Best-of-N (BoN) sampling with a reward model has been shown to be an effective strategy for aligning Large Language Models (LLMs) to human preferences at the time of decoding. BoN sampling is susceptible to a problem known as reward hacking when the accuracy of the reward model is not high enough due to the quality or the quantity of the preference dataset. Because the reward model is an imperfect proxy for the true objective, over-optimizing its value can compromise its performance on the true objective. In this research, we propose MBR-BoN, a variant of BoN that aims to mitigate reward hacking at inference time by incorporating the Minimum Bayes Risk (MBR) objective as a proximity regularization term. We show empirically and analytically that the MBR objective quantifies the proximity of the response to the reference policy, serving as a proximity regularizer. We evaluate MBR-BoN on the AlpacaFarm and Anthropic's hh-rlhf datasets and show that it outperforms both BoN sampling and MBR decoding. We also evaluate MBR-BoN to generate a pairwise preference learning dataset for Direct Preference Optimization (DPO). Empirical results show that models trained on a dataset generated with MBR-BoN outperform those with vanilla BoN. Our code is available at https://github.com/CyberAgentAILab/regularized-bon

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 57 of its 104 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

57 of 104 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  2. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  3. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  4. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  5. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  6. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  7. 2023Aligning Neural Machine Translation Models: Human Feedback in Training and Inference15 Nov
  8. 2023Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding14 Nov
  9. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  11. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  12. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 45

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2016Concrete Problems in AI Safetycited by 1,582
33 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  3. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  4. 2018Computational Optimal Transportcited by 765
  5. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  6. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  7. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  8. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  9. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  10. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  11. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  12. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  13. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  14. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  15. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 137
  16. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  17. 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
  18. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  19. 2021Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translationcited by 63
  20. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  21. 2021Facebook AI WMT21 News Translation Task Submissioncited by 57
  22. 2020Unbabel's Participation in the WMT20 Metrics Shared Taskcited by 56
  23. 2022Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decodingcited by 54
  24. 2023Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Informationcited by 52
  25. 2023It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Riskcited by 49
  26. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  27. 2023Faster Minimum Bayes Risk Decoding with Confidence-based Pruningcited by 34
  28. 2023MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methodscited by 33
  29. 2023This Land is {Your, My} Land: Evaluating Geopolitical Biases in Language Modelscited by 31
  30. 2024Linear-time Minimum Bayes Risk Decoding with Reference Aggregationcited by 21
  31. 2023DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4cited by 16
  32. 2024Hyperparameter-Free Approach for Faster Minimum Bayes Risk Decodingcited by 15
  33. 2023Quality-Aware Translation Models: Efficient Generation and Quality Estimation in a Single Modelcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 7; 2026: 11.

24 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  3. 2025Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Modelscited by 52
  4. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  5. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  6. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  7. 2024Test-Time Alignment via Hypothesis Reweightingcited by 11
  8. 2025Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanationscited by 9
  9. 2024HAF-RM: A Hybrid Alignment Framework for Reward Model Trainingcited by 5
  10. 2024M-Ped: Multi-Prompt Ensemble Decoding for Large Language Modelscited by 3
  11. 2025On the Role of Feedback in Test-Time Scaling of Agentic AI Workflowscited by 3
  12. 2025Inference Scaled GraphRAG: Improving Multi Hop Question Answering on Knowledge Graphscited by 2
12 more
  1. 2026Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignmentcited by 2
  2. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 2
  3. 2024An Actor-Critic Approach to Boosting Text-to-SQL Large Language Modelcited by 1
  4. 2025Ctrl-Z Sampling: Scaling Diffusion Sampling with Controlled Random Zigzag Explorationscited by 1
  5. 2026RSRCC: A Remote Sensing Regional Change Comprehension Benchmark Constructed via Retrieval-Augmented Best-of-N Rankingcited by 0
  6. 2026Joint Consistency: A Unified Test-Time Aggregation Framework via Energy Minimizationcited by 0
  7. 2026Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignmentcited by 0
  8. 2026GEOALIGN: Geometric Rollout Curation for Robust LLM Reinforcement Learningcited by 0
  9. 2026Best-of-Evidence: Best-of-N Selection under Partial Verificationcited by 0
  10. 2026Mitigating Over-Optimization in PRM-Guided Search in Mathematical Reasoning by Optimizing the Guidecited by 0
  11. 2026ExpBoN: Exponential-Noise Best-of-n for Efficient Test-Time LLM Alignmentcited by 0
  12. 2026Enhancing Small Language Models for Power Outage Report Generation via Minimum Risk Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignment
  2. 2025Soft Best-of-n Sampling for Model Alignment
  3. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling
  4. 2024Variational Best-of-N Alignment
  5. 2026Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignment
  6. 2025AdaBoN: Adaptive Best-of-N Alignment
  7. 2024Bayesian Reward Models for LLM Alignment
  8. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignment