mlsanityData to 14 Oct 2026

β-DPO: Direct Preference Optimization with Dynamic β

arXiv 2407.08639 ↗First posted 11 Jul 2024cs.AI cs.LGcited by 90

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) has emerged as a compelling approach for training Large Language Models (LLMs) to adhere to human preferences. However, the performance of DPO is sensitive to the fine-tuning of its trade-off parameter β, as well as to the quality of the preference data. We analyze the impact of β and data quality on DPO, uncovering that optimal β values vary with the informativeness of pairwise data. Addressing the limitations of static β values, we introduce a novel framework that dynamically calibrates β at the batch level, informed by data quality considerations. Additionally, our method incorporates β-guided data filtering to safeguard against the influence of outliers. Through empirical evaluation, we demonstrate that our dynamic β adjustment technique significantly improves DPO's performance across a range of models and datasets, offering a more robust and adaptable training paradigm for aligning LLMs with human feedback. The code is available at urlhttps://github.com/junkangwu/beta-DPO.

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 49 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

36 of 49 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization10 Jul
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024Filtered Direct Preference Optimization22 Apr
  4. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  6. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  7. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  8. 2023LIMA: Less Is More for Alignment18 May
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2015Continuous control with deep reinforcement learningcited by 3,407
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
15 more
  1. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  3. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  4. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  5. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  6. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  7. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  8. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  9. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
  10. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  11. 2023Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Datacited by 44
  12. 2021Temperature as Uncertainty in Contrastive Learningcited by 25
  13. 2024MolTC: Towards Molecular Relational Modeling In Language Modelscited by 22
  14. 2023Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasetscited by 19
  15. 2022Model-Aware Contrastive Learning: Towards Escaping the Dilemmascited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 52; 2026: 27.

90 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  2. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 83
  3. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 71
  4. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  5. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  6. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 29
  7. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 29
  8. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  9. 2024Unified Parameter-Efficient Unlearning for LLMscited by 25
  10. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  11. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 18
  12. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
38 more
  1. 2025Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimizationcited by 14
  2. 2025R.I.P.: Better Models by Survival of the Fittest Promptscited by 13
  3. 2025Robust Preference Optimization via Dynamic Target Marginscited by 13
  4. 2025Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systemscited by 12
  5. 2025What Matters in Data for DPO?cited by 12
  6. 2025Towards Transparent and Incentive-Compatible Collaboration in Decentralized LLM Multi-Agent Systems: A Blockchain-Driven Approachcited by 12
  7. 2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoningcited by 11
  8. 2025Teaching Your Models to Understand Code via Focal Preference Alignmentcited by 10
  9. 2024SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Maskscited by 9
  10. 2025AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Datasetcited by 9
  11. 2025In-context Ranking Preference Optimizationcited by 9
  12. 2025Preference Optimization by Estimating the Ratio of the Data Distributioncited by 9
  13. 2025LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Informationcited by 8
  14. 2025Thinking Preference Optimizationcited by 8
  15. 2024Systematic Reward Gap Optimization for Mitigating VLM Hallucinationscited by 7
  16. 2025Enhancing Reasoning Abilities of Small LLMs with Cognitive Alignmentcited by 7
  17. 2025Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimizationcited by 7
  18. 2025Gradient Imbalance in Direct Preference Optimizationcited by 6
  19. 2025AI Agent Behavioral Sciencecited by 6
  20. 2026PERSA: Reinforcement Learning for Professor-Style Personalized Feedback with LLMscited by 6
  21. 2024COS-DPO: Conditioned One-Shot Multi-Objective Fine-Tuning Frameworkcited by 5
  22. 2025Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Rankingcited by 5
  23. 2025KL Penalty Control via Perturbation for Direct Preference Optimizationcited by 5
  24. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPOcited by 5
  25. 2024REAL: Response Embedding-based Alignment for LLMscited by 4
  26. 2025RePO: Understanding Preference Learning Through ReLU-Based Optimizationcited by 4
  27. 2025ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models using Pareto High-quality Datacited by 4
  28. 2025Scalable Valuation of Human Feedback through Provably Robust Model Alignmentcited by 4
  29. 2025TB or Not TB: Coverage-Driven Direct Preference Optimization for Verilog Stimulus Generationcited by 4
  30. 2025Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigationcited by 4
  31. 2024Preference Alignment for Diffusion Model via Explicit Denoised Distribution Estimationcited by 3
  32. 2025Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomycited by 3
  33. 2025Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligningcited by 3
  34. 2025Fading to Grow: Growing Preference Ratios via Preference Fading Discrete Diffusion for Recommendationcited by 3
  35. 2025On Efficiency-Effectiveness Trade-off of Diffusion-based Recommenderscited by 3
  36. 2025Learning Noise-Resilient and Transferable Graph-Text Alignment via Dynamic Quality Assessmentcited by 3
  37. 2026DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucinationscited by 3
  38. 2025Understanding the Impact of Sampling Quality in Direct Preference Optimizationcited by 2

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Lightweight Robust Direct Preference Optimization
  2. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  3. 2024Direct Preference Optimization with an Offset
  4. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  5. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective
  6. 2025A Survey of Direct Preference Optimization
  7. 2024sDPO: Don't Use Your Data All at Once
  8. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model