mlsanityData to 7 Oct 2026

Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences

arXiv 2403.07230 ↗First posted 12 Mar 2024cs.CL cs.AI cs.LGcited by 39

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) is an effective technique that leverages pairwise preference data (usually one chosen and rejected response pair per user prompt) to align LLMs to human preferences. In practice, multiple responses can exist for a given prompt with varying quality relative to each other. With availability of such quality ratings for multiple responses, we propose utilizing these responses to create multiple preference pairs for a given prompt. Our work focuses on systematically using the constructed multiple preference pair in DPO training via curriculum learning methodology. In particular, we order these multiple pairs of preference data from easy to hard (emulating curriculum training) according to various criteria. We show detailed comparisons of our proposed approach to the standard single-pair DPO setting. Our method, which we call Curry-DPO consistently shows increased performance gains on MTbench, Vicuna, WizardLM, and the UltraFeedback test set, highlighting its effectiveness. More specifically, Curry-DPO achieves a score of 7.43 on MT-bench with Zephy-7B model outperforming majority of existing LLMs with similar parameter size. Curry-DPO also achieves the highest adjusted win rates on Vicuna, WizardLM, and UltraFeedback test datasets (90.7%, 87.1%, and 87.9% respectively) in our experiments, with notable gains of upto 7.5% when compared to standard DPO technique. We release the preference pairs used in alignment at: https://huggingface.co/datasets/ServiceNow-AI/Curriculum_DPO_preferences

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 51 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

25 of 51 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  2. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 20

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  11. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  12. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
8 more
  1. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  2. 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 201
  3. 2022ProsocialDialog: A Prosocial Backbone for Conversational Agentscited by 109
  4. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 71
  5. 2019Curriculum Learning for Domain Adaptation in Neural Machine Translationcited by 59
  6. 2019Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narrativescited by 59
  7. 2024Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrappingcited by 14
  8. 2021Exploiting Curriculum Learning in Unsupervised Neural Machine Translationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 21; 2026: 11.

39 papers.

Sort:

By area:

  1. 2025DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-trainingcited by 45
  2. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  3. 2025A Survey of Direct Preference Optimizationcited by 35
  4. 2025ExCoT: Optimizing Reasoning for Text-to-SQL with Execution Feedbackcited by 23
  5. 2024BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignmentcited by 20
  6. 2024Geometric-Averaged Preference Optimization for Soft Preference Labelscited by 18
  7. 2025Larger or Smaller Reward Margins to Select Preferences for Alignment?cited by 11
  8. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  9. 2024Multi-Reference Preference Optimization for Large Language Modelscited by 10
  10. 2025What Matters in Data for DPO?cited by 10
  11. 2025Value Drifts: Tracing Value Alignment During LLM Post-Trainingcited by 9
  12. 2024Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignmentcited by 7
27 more
  1. 2025Adaptive Helpfulness-Harmlessness Alignment with Preference Vectorscited by 6
  2. 2025LaMDAgent: An Autonomous Framework for Post-Training Pipeline Optimization via LLM Agentscited by 6
  3. 2024SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Marginscited by 5
  4. 2025REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspectivecited by 5
  5. 2025FRAME: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategycited by 4
  6. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignmentcited by 4
  7. 2025VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflectioncited by 4
  8. 2025Teaching with Lies: Curriculum DPO on Synthetic Negatives for Hallucination Detectioncited by 3
  9. 2026A Deep Dive into Scaling RL for Code Generation with Synthetic Data and Curriculacited by 3
  10. 2025Improving LLMs for Machine Translation Using Synthetic Preference Datacited by 2
  11. 2024On the Transformations across Reward Model, Parameter Update, and In-Context Promptcited by 1
  12. 2025Understanding the Impact of Sampling Quality in Direct Preference Optimizationcited by 1
  13. 2025From Fragments to Facts: A Curriculum-Driven DPO Approach for Generating Hindi News Veracity Explanationscited by 1
  14. 2025Learning More with Less: A Dynamic Dual-Level Down-Sampling Framework for Efficient Policy Optimizationcited by 1
  15. 2025Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spanscited by 1
  16. 2026CLewR: Curriculum Learning with Restarts for Machine Translation Preference Learningcited by 1
  17. 2026Entropy-Guided Data-Efficient Training for Multimodal Reasoning Reward Modelscited by 1
  18. 2026STAGE: Controlled Objective Admission for Multi-Preference LLM Alignmentcited by 1
  19. 2025Self-supervised Attribute-aware Dynamic Preference Ranking Alignmentcited by 0
  20. 2025g-DPO: Scalable Preference Optimization for Protein Language Modelscited by 0
  21. 2026SP^2DPO: An LLM-assisted Semantic Per-Pair DPO Generalizationcited by 0
  22. 2026Curriculum Group Policy Optimization: Adaptive Sampling for Unleashing the Potential of Text-to-Image Generationcited by 0
  23. 2026Curriculum Learning for Safety Alignmentcited by 0
  24. 2026Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimizationcited by 0
  25. 2026Disentangling Curriculum Learning in NLP: Towards a Unifying Taxonomycited by 0
  26. 2026What do Reward Models Memorize?cited by 0
  27. 2026AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization
  2. 2024Direct Preference Optimization with an Offset
  3. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts
  4. 2024Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization
  5. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts
  6. 2025Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm
  7. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
  8. 2024Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs