mlsanityData to 7 Oct 2026

Bootstrapping Language Models with DPO Implicit Rewards

arXiv 2406.09760 ↗First posted 14 Jun 2024cs.CL cs.LGcited by 53

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

Human alignment in large language models (LLMs) is an active area of research. A recent groundbreaking work, direct preference optimization (DPO), has greatly simplified the process from past work in reinforcement learning from human feedback (RLHF) by bypassing the reward learning stage in RLHF. DPO, after training, provides an implicit reward model. In this work, we make a novel observation that this implicit reward model can by itself be used in a bootstrapping fashion to further align the LLM. Our approach is to use the rewards from a current LLM to construct a preference dataset, which is then used in subsequent DPO rounds. We incorporate two refinements to further improve our approach: 1) length-regularized reward shaping to make the preference dataset length-unbiased; 2) experience replay to enhance the quality of the preference dataset. Our approach, named self-alignment with DPO ImpliCit rEwards (DICE), shows great improvements in alignment. It achieves an increase of more than 8% in lengthcontrolled win rate on AlpacaEval 2 for all the different base models that we tried, without relying on external feedback. Our code is available at https://github.com/sail-sg/dice.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 43 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

29 of 43 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  3. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  4. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  5. 2024Active Preference Learning for Large Language Models12 Feb
  6. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  8. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2018Experience Replay for Continual Learningcited by 1,074
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  7. 2022Large Language Models Can Self-Improvecited by 656
  8. 2024Self-Rewarding Language Modelscited by 654
  9. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  11. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  12. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
8 more
  1. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  2. 2024Iterative Reasoning Preference Optimizationcited by 244
  3. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  4. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  5. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  6. 2023Self-Alignment with Instruction Backtranslationcited by 149
  7. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  8. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 25; 2026: 17.

53 papers.

Sort:

By area:

  1. 2025Process Reinforcement through Implicit Rewardscited by 478
  2. 2025Reinforcing General Reasoning without Verifierscited by 76
  3. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  4. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 36
  5. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  6. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  7. 2024Selective Preference Optimization via Token-Level Reward Function Estimationcited by 25
  8. 2025Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewardscited by 23
  9. 2024Geometric-Averaged Preference Optimization for Soft Preference Labelscited by 18
  10. 2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 18
  11. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 17
  12. 2024Sample-Efficient Alignment for LLMscited by 16
38 more
  1. 2025Bag of Tricks for Inference-time Computation of LLM Reasoningcited by 15
  2. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillationcited by 15
  3. 2025MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Trainingcited by 12
  4. 2025Personalized LLM Decoding via Contrasting Personal Preferencecited by 12
  5. 2025MAPO: Mixed Advantage Policy Optimizationcited by 11
  6. 2025Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learningcited by 10
  7. 2025Uncovering the Impact of Chain-of-Thought Reasoning for Direct Preference Optimization: Lessons from Text-to-SQLcited by 9
  8. 2025Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignmentcited by 9
  9. 2024Length Desensitization in Direct Preference Optimizationcited by 8
  10. 2024Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learningcited by 7
  11. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  12. 2025Explicit Preference Optimization: No Need for an Implicit Reward Modelcited by 6
  13. 2024Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learningcited by 5
  14. 2024SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Marginscited by 5
  15. 2025REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspectivecited by 5
  16. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPOcited by 5
  17. 2025DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learningcited by 5
  18. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  19. 2024Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMscited by 4
  20. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimizationcited by 3
  21. 2026Reward Models Inherit Value Biases from Pretrainingcited by 3
  22. 2026SAIL: Self-Amplified Iterative Learning for Diffusion Model Alignment with Minimal Human Feedbackcited by 3
  23. 2024Dr. SoW: Density Ratio of Strong-over-weak LLMs for Reducing the Cost of Human Annotation in Preference Tuningcited by 1
  24. 2025Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Datacited by 1
  25. 2025Direct Advantage Regression: Aligning LLMs with Online AI Rewardcited by 1
  26. 2025CM-Align: Consistency-based Multilingual Alignment for Large Language Modelscited by 1
  27. 2025Coupled Variational Reinforcement Learning for Language Model General Reasoningcited by 1
  28. 2026Netflix Artwork Personalization via LLM Post-trainingcited by 1
  29. 2026GRLO: Towards Generalizable Reinforcement Learning in Open-Ended Environments from Zerocited by 1
  30. 2025DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)cited by 0
  31. 2025Bootstrapping LLMs via Preference-Based Policy Optimizationcited by 0
  32. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Modelscited by 0
  33. 2026IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucinationcited by 0
  34. 2026Bayesian Preference Learning for Test-Time Steerable Reward Modelscited by 0
  35. 2026A Lightweight Explainable Guardrail for Prompt Safetycited by 0
  36. 2026Se-DPO: Self-Evolving Token Credit for Direct Preference Optimizationcited by 0
  37. 2026Disentangling Optimization Scale from Preference Scale in DPOcited by 0
  38. 2026AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selectioncited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AlphaPO: Reward Shape Matters for LLM Alignment
  2. 2025Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
  3. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models
  4. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization
  5. 2025A Survey of Direct Preference Optimization
  6. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
  7. 2025Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms
  8. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization