mlsanityData to 7 Oct 2026

2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision

arXiv 2410.19720 ↗First posted 25 Oct 2024cs.CL cs.AIcited by 6

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Recent advancements in Direct Preference Optimization (DPO) have significantly enhanced the alignment of Large Language Models (LLMs) with human preferences, owing to its simplicity and effectiveness. However, existing methods typically optimize a scalar score or ranking reward, thereby overlooking the multi-dimensional nature of human preferences. In this work, we propose to extend the preference of DPO to two dimensions: segments and aspects. We first introduce a 2D supervision dataset called HelpSteer-2D. For the segment dimension, we divide the response into sentences and assign scores to each segment. For the aspect dimension, we meticulously design several criteria covering the response quality rubrics. With the 2-dimensional signals as feedback, we develop a 2D-DPO framework, decomposing the overall objective into multi-segment and multi-aspect objectives. Extensive experiments on popular benchmarks demonstrate that 2D-DPO performs better than methods that optimize for scalar or 1-dimensional preferences.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 46 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

30 of 46 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Selective Preference Optimization via Token-Level Reward Function Estimation24 Aug
  2. 2024Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level17 Jun
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  4. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  5. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  6. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  7. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  8. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment7 Nov
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  10. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  10. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  11. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  12. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
8 more
  1. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  2. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  3. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  4. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  5. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  6. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
  7. 2024Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraintcited by 39
  8. 2024Dynamic Multi-Reward Weighting for Multi-Style Controllable Generationcited by 13

Built on by

Cited by year

Citing papers per year: 2025: 6.

6 papers.

  1. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94
  2. 2025MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Trainingcited by 12
  3. 2025AIR: Complex Instruction Generation via Automatic Iterative Refinementcited by 10
  4. 2025DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Modelscited by 6
  5. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Modelscited by 6
  6. 2025Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigmcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025What Matters in Data for DPO?
  2. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective
  3. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forward
  4. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets
  5. 2024Direct Preference Optimization with an Offset
  6. 2025A Survey of Direct Preference Optimization
  7. 2026Autoregressive Direct Preference Optimization
  8. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts