mlsanityData to 7 Oct 2026

Robust LLM Alignment via Distributionally Robust Direct Preference Optimization

arXiv 2502.01930 ↗First posted 4 Feb 2025cs.LG cs.AIcited by 17

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

A major challenge in aligning large language models (LLMs) with human preferences is the issue of distribution shift. LLM alignment algorithms rely on static preference datasets, assuming that they accurately represent real-world user preferences. However, user preferences vary significantly across geographical regions, demographics, linguistic patterns, and evolving cultural trends. This preference distribution shift leads to catastrophic alignment failures in many real-world applications. We address this problem using the principled framework of distributionally robust optimization, and develop two novel distributionally robust direct preference optimization (DPO) algorithms, namely, Wasserstein DPO (WDPO) and Kullback-Leibler DPO (KLDPO). We characterize the sample complexity of learning the optimal policy parameters for WDPO and KLDPO. Moreover, we propose scalable gradient descent-style learning algorithms by developing suitable approximations for the challenging minimax loss functions of WDPO and KLDPO. Our empirical experiments using benchmark data sets and LLMs demonstrate the superior performance of WDPO and KLDPO in substantially improving the alignment when there is a preference distribution shift.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 76 references that are in the map. First posted on arXiv 4 Feb 2025.

Builds on

50 of 76 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization18 Jul
  2. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  3. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization10 Jul
  4. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  5. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Models17 Oct
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  12. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
32 more
  1. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  2. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  3. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  4. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  5. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
  6. 2015Data-driven Distributionally Robust Optimization Using the Wasserstein Metric: Performance Guarantees and Tractable Reformulationscited by 285
  7. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  8. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  9. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  10. 2016Distributionally Robust Stochastic Optimization with Wasserstein Distancecited by 182
  11. 2020Large-Scale Methods for Distributionally Robust Optimizationcited by 179
  12. 2019Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learningcited by 169
  13. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
  14. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  15. 2019Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensemblescited by 100
  16. 2022Robust Reinforcement Learning using Offline Datacited by 95
  17. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  18. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 85
  19. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
  20. 2017Regularization via Mass Transportationcited by 79
  21. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
  22. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  23. 2021Distributionally Robust Learningcited by 44
  24. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  25. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 35
  26. 2024Reward-Robust RLHF in LLMscited by 32
  27. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  28. 2024Robust Reinforcement Learning from Corrupted Human Feedbackcited by 25
  29. 2023A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shiftcited by 17
  30. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coveragecited by 12
  31. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularizationcited by 11
  32. 2023Bring Your Own (Non-Robust) Algorithm to Solve Robust MDPs by Estimating The Worst Kernelcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 11.

17 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  3. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  4. 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimizationcited by 16
  5. 2026Revisiting Robustness for LLM Safety Alignment via Selective Geometry Controlcited by 3
  6. 2025Robust Preference Alignment via Directional Neighborhood Consensuscited by 2
  7. 2026Objective Decoupling in Social Reinforcement Learning: Recovering Ground Truth from Sycophantic Majoritiescited by 2
  8. 2026Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judgecited by 2
  9. 2026DeDPO: Debiased Direct Preference Optimization for Diffusion Modelscited by 1
  10. 2026Mitigating Mismatch within Reference-based Preference Optimizationcited by 1
  11. 2026Oracle-Robust Online Alignment for Large Language Modelscited by 1
  12. 2026Cat-DPO: Category-Adaptive Safety Alignmentcited by 1
5 more
  1. 2025Preference Robustness for DPO with Applications to Public Healthcited by 0
  2. 2026Distributionally Robust Listwise Preference Optimizationcited by 0
  3. 2026FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experiencecited by 0
  4. 2026Mathematical Programming in Machine Learning and Artificial Intelligence: A Unified Taxonomy of Models and Applicationscited by 0
  5. 2026Robust Nash Alignment under Preference Uncertaintycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
  2. 2026A Zeroth-Order Paradigm for LLM Preference Alignment
  3. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
  4. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  5. 2025ComPO: Preference Alignment via Comparison Oracles
  6. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  7. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  8. 2025SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment