mlsanityData to 7 Oct 2026

Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment

arXiv 2412.14516 ↗First posted 19 Dec 2024cs.LG cs.CLcited by 50

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We study the problem of aligning large language models (LLMs) with human preference data. Contrastive preference optimization has shown promising results in aligning LLMs with available preference data by optimizing the implicit reward associated with the policy. However, the contrastive objective focuses mainly on the relative values of implicit rewards associated with two responses while ignoring their actual values, resulting in suboptimal alignment with human preferences. To address this limitation, we propose calibrated direct preference optimization (Cal-DPO), a simple yet effective algorithm. We show that substantial improvement in alignment with the given preferences can be achieved simply by calibrating the implicit reward to ensure that the learned implicit rewards are comparable in scale to the ground-truth rewards. We demonstrate the theoretical advantages of Cal-DPO over existing approaches. The results of our experiments on a variety of standard benchmarks show that Cal-DPO remarkably improves off-the-shelf methods.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 73 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

56 of 73 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective14 Oct
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  4. 2024Token-level Direct Preference Optimization18 Apr
  5. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  6. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective6 Apr
  7. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  8. 2024COPR: Continual Human Preference Learning via Optimal Policy Regularization22 Feb
  9. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  10. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  11. 2024Noise Contrastive Alignment of Language Models with Explicit Rewards8 Feb
  12. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
6 more
  1. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  4. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  5. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignment4 Jun
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
26 more
  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  4. 2024Self-Rewarding Language Modelscited by 654
  5. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  6. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  7. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  8. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  9. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  10. 2024Iterative Reasoning Preference Optimizationcited by 244
  11. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  12. 2023Nash Learning from Human Feedbackcited by 220
  13. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  14. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  15. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  16. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  17. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  18. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
  19. 2016Reward Augmented Maximum Likelihood for Neural Structured Predictioncited by 80
  20. 2023On the Calibration of Large Language Models and Alignmentcited by 72
  21. 2019Learning to Generalize from Sparse and Underspecified Rewardscited by 39
  22. 2023Policy Optimization in RLHF: The Impact of Out-of-preference Datacited by 35
  23. 2019Calibrated Model-Based Deep Reinforcement Learningcited by 31
  24. 2023Learning How to Propagate Messages in Graph Neural Networkscited by 23
  25. 2023A General Offline Reinforcement Learning Framework for Interactive Recommendationcited by 18
  26. 2016Improving Policy Gradient by Exploring Under-appreciated Rewardscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 30; 2026: 20.

50 papers.

Sort:

By area:

  1. 2025A Survey of Direct Preference Optimizationcited by 35
  2. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameterscited by 25
  3. 2025On a Connection Between Imitation Learning and RLHFcited by 13
  4. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  5. 2025Larger or Smaller Reward Margins to Select Preferences for Alignment?cited by 11
  6. 2025ComPO: Preference Alignment via Comparison Oraclescited by 11
  7. 2025OmniInsert: Mask-Free Video Insertion of Any Reference via Diffusion Transformer Modelscited by 9
  8. 2025Incentivizing Strong Reasoning from Weak Supervisioncited by 8
  9. 2025Inference-time Alignment in Continuous Spacecited by 7
  10. 2025SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Modelscited by 7
  11. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Modelscited by 6
  12. 2025Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Modelscited by 6
38 more
  1. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPOcited by 5
  2. 2026TAB-PO: Preference Optimization with a Token-Level Adaptive Barrier for Token-Critical Structured Generationcited by 5
  3. 2026QualiTeacher: Quality-Conditioned Pseudo-Labeling for Real-World Image Restorationcited by 5
  4. 2025DreaMontage: Arbitrary Frame-Guided One-Shot Video Generationcited by 4
  5. 2026CATTO: Balancing Preferences and Confidence in Language Modelscited by 4
  6. 2025The Differences Between Direct Alignment Algorithms are a Blurcited by 3
  7. 2025SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignmentcited by 3
  8. 2025Beyond RLHF: A Unified Theoretical Framework of Alignmentcited by 3
  9. 2025CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferencescited by 3
  10. 2025C2-DPO: Constrained Controlled Direct Preference Optimizationcited by 2
  11. 2025Leveraging Machine Unlearning for Cost-Efficient Preference Alignmentcited by 2
  12. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?cited by 2
  13. 2025AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignmentcited by 2
  14. 2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gainscited by 2
  15. 2026Small-Margin Preferences Still Matter-If You Train Them Rightcited by 2
  16. 2025Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimizationcited by 1
  17. 2025OraPO: Oracle-educated Reinforcement Learning for Data-efficient and Factual Radiology Report Generationcited by 1
  18. 2025LLM4Rec: Large Language Models for Multimodal Generative Recommendation with Causal Debiasingcited by 1
  19. 2025Offline Preference Optimization via Maximum Marginal Likelihood Estimationcited by 1
  20. 2026Autoregressive Direct Preference Optimizationcited by 1
  21. 2026PVminerLLM: Structured Extraction of Patient Voice from Patient-Generated Text using Large Language Modelscited by 1
  22. 2026CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignmentcited by 1
  23. 2026Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimizationcited by 1
  24. 2026Gradient-Gated DPO: Stabilizing Preference Optimization in Language Modelscited by 1
  25. 2026AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updatescited by 1
  26. 2025BPO: Revisiting Preference Modeling in Direct Preference Optimizationcited by 0
  27. 2025On Monotonicity in AI Alignmentcited by 0
  28. 2025Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineeringcited by 0
  29. 2025SSFO: Self-Supervised Faithfulness Optimization for Retrieval-Augmented Generationcited by 0
  30. 2026Displacement-Resistant Extensions of DPO with Nonconvex f-Divergencescited by 0
  31. 2026IRIS: Interpolative R'enyi Iterative Self-play for Large Language Model Fine-Tuningcited by 0
  32. 2026Fusion in Your Way: Aligning Image Fusion with Heterogeneous Demands via Direct Preference Optimizationcited by 0
  33. 2026Distributed Direct Preference Optimizationcited by 0
  34. 2026PVminerLLM2: Improving Structured Extraction of Patient Voice via Preference Optimizationcited by 0
  35. 2026Antigen-specific Antibody Multi-modal Foundation Model for Functional Antibody Designcited by 0
  36. 2026Context Blindness in DPO: Mitigating Object Hallucination in MLLMs via Context-Calibrated Preference Optimizationcited by 0
  37. 2026A Zeroth-Order Paradigm for LLM Preference Alignmentcited by 0
  38. 2026MASCRDM: Multi-Agent System for Compliance Risk Detection and Mitigation in Training Process of Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Efficient Exact Optimization of Language Model Alignment
  2. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  3. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization
  4. 2026BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment
  5. 2024sDPO: Don't Use Your Data All at Once
  6. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  7. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts
  8. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks