mlsanityData to 14 Oct 2026

Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm

arXiv 2505.01706 ↗First posted 3 May 2025cs.AI cs.CL cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimisation (DPO) has emerged as a powerful method for aligning Large Language Models (LLMs) with human preferences, offering a stable and efficient alternative to approaches that use Reinforcement learning via Human Feedback. In this work, we investigate the performance of DPO using open-source preference datasets. One of the major drawbacks of DPO is that it doesn't induce granular scoring and treats all the segments of the responses with equal propensity. However, this is not practically true for human preferences since even "good" responses have segments that may not be preferred by the annotator. To resolve this, a 2-dimensional scoring for DPO alignment called 2D-DPO was proposed. We explore the 2D-DPO alignment paradigm and the advantages it provides over the standard DPO by comparing their win rates. It is observed that these methods, even though effective, are not robust to label/score noise. To counter this, we propose an approach of incorporating segment-level score noise robustness to the 2D-DPO algorithm. Along with theoretical backing, we also provide empirical verification in favour of the algorithm and introduce other noise models that can be present.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 16 of its 21 references that are in the map. First posted on arXiv 3 May 2025.

Builds on

16 of 21 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025A Survey of Direct Preference Optimization12 Mar
  2. 2025Diverse Preference Optimization30 Jan
  3. 20242D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision25 Oct
  4. 2024Token-level Direct Preference Optimization18 Apr
  5. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  6. 2023Large Language Model Alignment: A Survey26 Sep

Other references · 10

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  7. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  8. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 256
  9. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 74
  10. 2022Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedbackcited by 26

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Direct Preference Optimization with an Offset
  2. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization
  3. 2025Lightweight Robust Direct Preference Optimization
  4. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  5. 2025What Matters in Data for DPO?
  6. 2024Minor DPO reject penalty to increase training robustness
  7. 2025Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
  8. 2025Less is More: Improving LLM Alignment via Preference Data Selection