Data to 7 Oct 2026

Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization

arXiv 2410.08847 ↗First posted 11 Oct 2024cs.LG cs.AI cs.CLcited by 66

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) and its variants are increasingly used for aligning language models with human preferences. Although these methods are designed to teach a model to generate preferred responses more frequently relative to dispreferred responses, prior work has observed that the likelihood of preferred responses often decreases during training. The current work sheds light on the causes and implications of this counter-intuitive phenomenon, which we term likelihood displacement. We demonstrate that likelihood displacement can be catastrophic, shifting probability mass from preferred responses to responses with an opposite meaning. As a simple example, training a model to prefer textttNo over textttNever can sharply increase the probability of textttYes. Moreover, when aligning the model to refuse unsafe prompts, we show that such displacement can unintentionally lead to unalignment, by shifting probability mass from preferred refusal responses to harmful responses (e.g., reducing the refusal rate of Llama-3-8B-Instruct from 74.4% to 33.4%). We theoretically characterize that likelihood displacement is driven by preferences that induce similar embeddings, as measured by a centered hidden embedding similarity (CHES) score. Empirically, the CHES score enables identifying which training samples contribute most to likelihood displacement in a given dataset. Filtering out these samples effectively mitigated unintentional unalignment in our experiments. More broadly, our results highlight the importance of curating data with sufficiently distinct preferences, for which we believe the CHES score may prove valuable.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 64 of its 72 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

64 of 72 references.

Closest ancestors · 19

Same direction, newest first

7 more

Other references · 45

2017Attention Is All You Needcited by 59,932
2023GPT-4 Technical Reportcited by 18,132
2024The Llama 3 Herd of Modelscited by 14,192
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
2020Learning to summarize from human feedbackcited by 2,562
2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
2019Fine-Tuning Language Models from Human Preferencescited by 2,124
2024Mixtral of Expertscited by 1,785
2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106

33 more

2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
2024OLMo: Accelerating the Science of Language Modelscited by 599
2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
2024Iterative Reasoning Preference Optimizationcited by 244
2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
2020Neural collapse with unconstrained featurescited by 140
2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 95
2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
2020A Mathematical Exploration of Why Language Models Help Solve Downstream Taskscited by 71
2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
2022Perturbation Analysis of Neural Collapsecited by 34
2020A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscited by 31
2023Exploiting Novel GPT-4 APIscited by 27
2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 26
2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 40; 2026: 25.

66 papers.

Sort:

By area:

2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
2025Can Large Reasoning Models Self-Train?cited by 66
2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
2025A Survey of Direct Preference Optimizationcited by 35
2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
2025On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimizationcited by 31
2025Training a Generally Curious Agentcited by 29
2025The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gainscited by 25
2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 25
2025ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergencecited by 21
2025Conformal Linguistic Calibration: Trading-off between Factuality and Specificitycited by 20
2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 18

38 more

2025MPO: Multilingual Safety Alignment via Reward Gap Optimizationcited by 18
2025LLM Unlearning with LLM Beliefscited by 17
2025On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacementcited by 15
2025AlphaPO: Reward Shape Matters for LLM Alignmentcited by 14
2025ComPO: Preference Alignment via Comparison Oraclescited by 11
2025Towards Safe Reasoning in Large Reasoning Models via Corrective Interventioncited by 11
2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
2026Where does output diversity collapse in post-training?cited by 10
2025Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learningcited by 9
2025Behavior Injection: Preparing Language Models for Reinforcement Learningcited by 8
2025Sequences of Logits Reveal the Low Rank Structure of Language Modelscited by 8
2026Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Modelscited by 8
2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewardscited by 8
2025Post-training for Efficient Communication via Convention Formationcited by 7
2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
2025Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functionscited by 6
2024Understanding the Logic of Direct Preference Alignment through Logiccited by 5
2025SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learningcited by 5
2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPOcited by 5
2026Causal Scene Narration with Runtime Safety Supervision for Vision-Language-Action Drivingcited by 5
2025Preference learning made easy: Everything should be understood through win ratecited by 3
2025SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignmentcited by 3
2025Why DPO is a Misspecified Estimator and How to Fix Itcited by 3
2025C2-DPO: Constrained Controlled Direct Preference Optimizationcited by 2
2026Learning Dynamics in RL Post-Training for Language Modelscited by 2
2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gainscited by 2
2026Small-Margin Preferences Still Matter-If You Train Them Rightcited by 2
2026The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Creditscited by 2
2025PIPA: Preference Alignment as Prior-Informed Statistical Estimationcited by 1
2025The Limits of Preference Data for Post-Trainingcited by 1
2025Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimizationcited by 1
2025LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMscited by 1
2025Generalizing while preserving monotonicity in comparison-based preference learning modelscited by 1
2026Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimizationcited by 1
2026ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraintscited by 1
2026Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVRcited by 1
2026Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?cited by 1
2026Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winnercited by 1

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way