Data to 7 Oct 2026

Provably Robust DPO: Aligning Language Models with Noisy Feedback

arXiv 2403.00409 ↗First posted 1 Mar 2024cs.LG cs.CLcited by 133

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Learning from preference-based feedback has recently gained traction as a promising approach to align language models with human interests. While these aligned generative models have demonstrated impressive capabilities across various tasks, their dependence on high-quality human preference data poses a bottleneck in practical applications. Specifically, noisy (incorrect and ambiguous) preference pairs in the dataset might restrict the language models from capturing human intent accurately. While practitioners have recently proposed heuristics to mitigate the effect of noisy preferences, a complete theoretical understanding of their workings remain elusive. In this work, we aim to bridge this gap by by introducing a general framework for policy optimization in the presence of random preference flips. We focus on the direct preference optimization (DPO) algorithm in particular since it assumes that preferences adhere to the Bradley-Terry-Luce (BTL) model, raising concerns about the impact of noisy data on the learned policy. We design a novel loss function, which de-bias the effect of noise on average, making a policy trained by minimizing that loss robust to the noise. Under log-linear parameterization of the policy class and assuming good feature coverage of the SFT policy, we prove that the sub-optimality gap of the proposed robust DPO (rDPO) policy compared to the optimal policy is of the order O(frac11-2ε√fracdn), where ε < 1/2 is flip rate of labels, d is policy parameter dimension and n is size of dataset. Our experiments on IMDb sentiment generation and Anthropic's helpful-harmless dataset show that rDPO is robust to noise in preference labels compared to vanilla DPO and other heuristics proposed by practitioners.

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 36 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

24 of 36 references.

Closest ancestors · 3

Same direction, newest first

Other references · 21

2017Proximal Policy Optimization Algorithmscited by 14,441
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2017Deep reinforcement learning from human preferencescited by 3,846
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2020Learning to summarize from human feedbackcited by 2,562
2019When Does Label Smoothing Help?cited by 985
2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
2015An Introduction to Matrix Concentration Inequalitiescited by 391
2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244

9 more

2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
2022Offline Reinforcement Learning with Differential Privacycited by 19
2023Differentially Private Reward Estimation with Preference Feedbackcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 34; 2025: 60; 2026: 39.

133 papers.

Sort:

By area:

2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
2024Vision-Language Models Can Self-Improve Reasoning via Reflectioncited by 66
2025Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyondcited by 59
2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimizationcited by 52
2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
2025Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuningcited by 39
2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 38
2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
2025A Survey of Direct Preference Optimizationcited by 35

38 more

2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 31
2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 24
2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimizationcited by 19
2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
2024Geometric-Averaged Preference Optimization for Soft Preference Labelscited by 18
2025MPO: Multilingual Safety Alignment via Reward Gap Optimizationcited by 18
2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 17
2024ROPO: Robust Preference Optimization for Large Language Modelscited by 16
2025Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimizationcited by 14
2024Reward Learning From Preference With Tiescited by 13
2024Towards Acyclic Preference Evaluation of Language Models via Multiple Evaluatorscited by 13
2024T-REG: Preference Optimization with Token-Level Reward Regularizationcited by 13
2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
2025Robust Preference Optimization via Dynamic Target Marginscited by 13
2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 12
2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
2025Doubly Robust Alignment for Large Language Modelscited by 11
2024Reward Difference Optimization For Sample Reweighting In Offline RLHFcited by 10
2024Soft Preference Optimization: Aligning Language Models to Expert Distributionscited by 9
2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Onlycited by 9
2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignmentcited by 9
2025Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Timecited by 8
2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPOcited by 8
2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7
2024VideoSAVi: Self-Aligned Video Language Models without Human Supervisioncited by 7
2025From Prompting to Alignment: A Generative Framework for Query Recommendationcited by 7
2025Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimizationcited by 7
2024PORT: Preference Optimization on Reasoning Tracescited by 6
2025Gradient Imbalance in Direct Preference Optimizationcited by 6
2025Improving Protein Sequence Design through Designability Preference Optimizationcited by 6
2025Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Taskscited by 6
2025Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Modelscited by 6
2026CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbackscited by 6
2024Alignment with Preference Optimization Is All You Need for LLM Safetycited by 5
2024Challenges and Future Directions of Data-Centric AI Alignmentcited by 5
2024Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Bothcited by 5

50 most cited of 133 All 133 (JSON)

Similar, not cited

Close in wording, no citation either way