mlsanityData to 7 Oct 2026

Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level

arXiv 2406.11817 ↗First posted 17 Jun 2024cs.CL cs.AI cs.LGcited by 22

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO), a standard method for aligning language models with human preferences, is traditionally applied to offline preferences. Recent studies show that DPO benefits from iterative training with online preferences labeled by a trained reward model. In this work, we identify a pitfall of vanilla iterative DPO - improved response quality can lead to increased verbosity. To address this, we introduce iterative length-regularized DPO (iLR-DPO) to penalize response length. Our empirical results show that iLR-DPO can enhance a 7B model to perform on par with GPT-4 without increasing verbosity. Specifically, our 7B model achieves a 50.5% length-controlled win rate against textttGPT-4 Preview on AlpacaEval 2.0, and excels across standard benchmarks including MT-Bench, Arena-Hard and OpenLLM Leaderboard. These results demonstrate the effectiveness of iterative DPO in aligning language models with human feedback.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 14 of its 19 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

14 of 19 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Robust Preference Optimization through Reward Model Distillation29 May
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  4. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  5. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 8

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  6. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  8. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 12; 2026: 2.

22 papers.

Sort:

By area:

  1. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  2. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  3. 2024GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Modelscited by 42
  4. 2025DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Drivingcited by 39
  5. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 36
  6. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedbackcited by 31
  7. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  8. 2024I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigmcited by 27
  9. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  10. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  11. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrastscited by 9
  12. 20242D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervisioncited by 6
10 more
  1. 2025Reward Models Identify Consistency, Not Causalitycited by 6
  2. 2025DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Modelscited by 6
  3. 2025AMPO: Active Multi-Preference Optimization for Self-play Preference Selectioncited by 5
  4. 2025REA-RL: Reflection-Aware Online Reinforcement Learning for Efficient Reasoningcited by 5
  5. 2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 5
  6. 2025Beyond RLHF: A Unified Theoretical Framework of Alignmentcited by 3
  7. 2025CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignmentcited by 2
  8. 2025Is On-Policy Data always the Best Choice for Direct Preference Optimization-based LM Alignment?cited by 1
  9. 2025Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignmentcited by 1
  10. 2026GRLO: Towards Generalizable Reinforcement Learning in Open-Ended Environments from Zerocited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Direct Preference Optimization
  2. 2024Length Desensitization in Direct Preference Optimization
  3. 2024Direct Preference Optimization with an Offset
  4. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective
  5. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  6. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets
  7. 2024New Desiderata for Direct Preference Optimization
  8. 2024AIPO: Improving Training Objective for Iterative Preference Optimization