mlsanityData to 7 Oct 2026

Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization

arXiv 2407.07880 ↗First posted 10 Jul 2024cs.LG cs.AI cs.CLcited by 52

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

This study addresses the challenge of noise in training datasets for Direct Preference Optimization (DPO), a method for aligning Large Language Models (LLMs) with human preferences. We categorize noise into pointwise noise, which includes low-quality data points, and pairwise noise, which encompasses erroneous data pair associations that affect preference rankings. Utilizing Distributionally Robust Optimization (DRO), we enhance DPO's resilience to these types of noise. Our theoretical insights reveal that DPO inherently embeds DRO principles, conferring robustness to pointwise noise, with the regularization coefficient β playing a critical role in its noise resistance. Extending this framework, we introduce Distributionally Robustifying DPO (Dr. DPO), which integrates pairwise robustness by optimizing against worst-case pairwise scenarios. The novel hyperparameter β' in Dr. DPO allows for fine-tuned control over data pair reliability, providing a strategic balance between exploration and exploitation in noisy training environments. Empirical evaluations demonstrate that Dr. DPO substantially improves the quality of generated text and response accuracy in preference datasets, showcasing enhanced performance in both noisy and noise-free settings. The code is available at https://github.com/junkangwu/DrDPO.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 50 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

35 of 50 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  2. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 217 Nov
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  4. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  5. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2021Convex optimizationcited by 2,479
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2023Towards Understanding Sycophancy in Language Modelscited by 988
17 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  2. 2023Textbooks Are All You Needcited by 561
  3. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  4. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  5. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
  6. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 333
  7. 2016Variance-based regularization with convex objectivescited by 153
  8. 2015Distributionally Robust Logistic Regressioncited by 124
  9. 2021DORO: Distributional and Outlier Robust Optimizationcited by 54
  10. 2021Modeling the Second Player in Distributionally Robust Optimizationcited by 29
  11. 2019Distributionally Robust Counterfactual Risk Minimizationcited by 26
  12. 2022Distributionally Robust Models with Parametric Likelihood Ratioscited by 17
  13. 2020Sampling-Decomposable Generative Adversarial Recommendercited by 16
  14. 2021Label Distributionally Robust Losses for Multi-class Classification: Consistency, Robustness and Adaptivitycited by 11
  15. 2017The discrete moment problem with nonconvex shape constraintscited by 6
  16. 2022Coresets for Wasserstein Distributionally Robust Optimization Problemscited by 6
  17. 2021Orthounimodal Distributionally Robust Optimization: Representation, Computation and Multivariate Extreme Event Applicationscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 24; 2026: 23.

52 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 89
  3. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  4. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 38
  5. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  6. 2025A Survey of Direct Preference Optimizationcited by 35
  7. 2024No Preference Left Behind: Group Distributional Preference Optimizationcited by 21
  8. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 17
  9. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  10. 2025Robust Preference Optimization via Dynamic Target Marginscited by 13
  11. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 12
  12. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
38 more
  1. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  2. 2025Doubly Robust Alignment for Large Language Modelscited by 11
  3. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMscited by 9
  4. 2025VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Trainingcited by 9
  5. 2024Rethinking the Uncertainty: A Critical Review and Analysis in the Era of Large Language Modelscited by 6
  6. 2026CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbackscited by 6
  7. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 5
  8. 2025RePO: Understanding Preference Learning Through ReLU-Based Optimizationcited by 4
  9. 2025Scalable Valuation of Human Feedback through Provably Robust Model Alignmentcited by 4
  10. 2025On Symmetric Losses for Robust Policy Optimization with Noisy Preferencescited by 4
  11. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignmentcited by 4
  12. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  13. 2026Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Trainingcited by 4
  14. 2025How Well Can Preference Optimization Generalize Under Noisy Feedback?cited by 3
  15. 2026Revisiting Robustness for LLM Safety Alignment via Selective Geometry Controlcited by 3
  16. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shiftscited by 2
  17. 2025Lightweight Robust Direct Preference Optimizationcited by 2
  18. 2025Distributionally Robust Optimization with Adversarial Data Contaminationcited by 1
  19. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 1
  20. 2025VLIC: Vision-Language Models As Perceptual Judges for Human-Aligned Image Compressioncited by 1
  21. 2026DeDPO: Debiased Direct Preference Optimization for Diffusion Modelscited by 1
  22. 2026Mitigating Mismatch within Reference-based Preference Optimizationcited by 1
  23. 2026Oracle-Robust Online Alignment for Large Language Modelscited by 1
  24. 2026Cat-DPO: Category-Adaptive Safety Alignmentcited by 1
  25. 2025Online Distributionally Robust LLM Alignment via Regression to Relative Rewardcited by 0
  26. 2026Region-Normalized DPO for Medical Image Segmentation under Noisy Judgescited by 0
  27. 2026Learning Where It Matters: Geometric Anchoring for Robust Preference Alignmentcited by 0
  28. 2026DARC: Disagreement-Aware Alignment via Risk-Constrained Decodingcited by 0
  29. 2026Causal Direct Preference Optimization for Distributionally Robust Generative Recommendationcited by 0
  30. 2026Distributionally Robust Token Optimization in RLHFcited by 0
  31. 2026Efficient Preference Poisoning Attack on Offline RLHFcited by 0
  32. 2026Distributionally Robust Set Representation Learning Under Inference-Time Element Corruptioncited by 0
  33. 2026The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoningcited by 0
  34. 2026When Good Verifiers Go Bad: Silent Negative Transfer in Verifier-Guided VLM Trainingcited by 0
  35. 2026Distributionally Robust Listwise Preference Optimizationcited by 0
  36. 2026Unbiased Alignment for Large Language Models with Noisy Preferencescited by 0
  37. 2026Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labelscited by 0
  38. 2026Positive-Unlabeled Preference Optimization For Chest X-ray Report Generationcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ROPO: Robust Preference Optimization for Large Language Models
  2. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  3. 2025Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm
  4. 2026BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment
  5. 2024Noise Contrastive Alignment of Language Models with Explicit Rewards
  6. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
  7. 2025ComPO: Preference Alignment via Comparison Oracles
  8. 2026Private Direct Preference Optimization for LLM Alignment