mlsanityData to 7 Oct 2026

sDPO: Don't Use Your Data All at Once

arXiv 2403.19270 ↗First posted 28 Mar 2024cs.CL cs.AIcited by 46

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

As development of large language models (LLM) progresses, aligning them with human preferences has become increasingly important. We propose stepwise DPO (sDPO), an extension of the recently popularized direct preference optimization (DPO) for alignment tuning. This approach involves dividing the available preference datasets and utilizing them in a stepwise manner, rather than employing it all at once. We demonstrate that this method facilitates the use of more precisely aligned reference models within the DPO training framework. Furthermore, sDPO trains the final model to be more performant, even outperforming other popular LLMs with more parameters.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 30 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

19 of 30 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 217 Nov

Other references · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
6 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  3. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  5. 2021Scaling Laws for Transfercited by 265
  6. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 18; 2026: 5.

46 papers.

Sort:

By area:

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  2. 2024A Survey on Large Language Models for Code Generationcited by 646
  3. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  4. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  5. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 67
  6. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  7. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  8. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  9. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  10. 2024Stress-Testing Capability Elicitation With Password-Locked Modelscited by 42
  11. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  12. 2025A Survey of Direct Preference Optimizationcited by 35
34 more
  1. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  2. 2024Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arenacited by 26
  3. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  4. 2025GRPO-λ: Credit Assignment improves LLM Reasoningcited by 21
  5. 2025Rejected Dialects: Biases Against African American Language in Reward Modelscited by 20
  6. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimizationcited by 19
  7. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  8. 2025MagicID: Hybrid Preference Optimization for ID-Consistent and Dynamic-Preserved Video Customizationcited by 17
  9. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 16
  10. 2024Octopus v4: Graph of language modelscited by 11
  11. 2024UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Functioncited by 9
  12. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrastscited by 9
  13. 2025PerPO: Perceptual Preference Optimization via Discriminative Rewardingcited by 9
  14. 2025Aligning Large Language Models with Implicit Preferences from User-Generated Contentcited by 9
  15. 2024Permutative Preference Alignment from Listwise Ranking of Human Judgmentscited by 7
  16. 2025iTool: Reinforced Fine-Tuning with Dynamic Deficiency Calibration for Advanced Tool Usecited by 7
  17. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimizationcited by 6
  18. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Modelcited by 6
  19. 2025Adaptive Helpfulness-Harmlessness Alignment with Preference Vectorscited by 6
  20. 2024AIPO: Improving Training Objective for Iterative Preference Optimizationcited by 5
  21. 2024SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Marginscited by 5
  22. 2025AMPO: Active Multi-Preference Optimization for Self-play Preference Selectioncited by 5
  23. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularizationcited by 4
  24. 2025RePO: Understanding Preference Learning Through ReLU-Based Optimizationcited by 4
  25. 2025Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settingscited by 3
  26. 2024Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimizationcited by 1
  27. 2025Understanding the Impact of Sampling Quality in Direct Preference Optimizationcited by 1
  28. 2025MotionFlux: Efficient Text-Guided Motion Generation through Rectified Flow Matching and Preference Alignmentcited by 1
  29. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  30. 2024An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problemscited by 0
  31. 2026Latent Adversarial Regularization for Offline Preference Optimizationcited by 0
  32. 2026Decoupling Strategy and Execution in Task-Focused Dialogue via Goal-Oriented Preference Optimizationcited by 0
  33. 2026S-SPPO: Semantic-Calibrated Self-Play Preference Optimizationcited by 0
  34. 2026Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
  2. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  3. 2024Direct Preference Optimization with an Offset
  4. 2025Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information
  5. 2025SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment
  6. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  7. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  8. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks