mlsanityData to 7 Oct 2026

Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimization

arXiv 2603.18258 ↗First posted 18 Mar 2026cs.LG cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) has emerged as a popular algorithm for aligning pretrained large language models with human preferences, owing to its simplicity and training stability. However, DPO suffers from the recently identified squeezing effect (also known as likelihood displacement), where the probability of preferred responses decreases unintentionally during training. To understand and mitigate this phenomenon, we develop a theoretical framework that models the coordinate-wise dynamics in logit space. Our analysis reveals that negative-gradient updates cause residuals to expand rapidly along high-curvature directions, which underlies the squeezing effect, whereas Sharpness-Aware Minimization (SAM) can suppress this behavior through its curvature-regularization effect. Building on this insight, we investigate logits-SAM, a computationally efficient variant that perturbs only the output layer with negligible overhead. Extensive experiments on Pythia-2.8B, Mistral-7B, and Gemma-2B-IT across multiple datasets and benchmarks demonstrate that logits-SAM consistently improves the effectiveness of DPO and integrates seamlessly with other DPO variants. Code is available at https://github.com/RitianLuo/logits-sam-dpo.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 71 references that are in the map. First posted on arXiv 18 Mar 2026.

Builds on

56 of 71 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025C2-DPO: Constrained Controlled Direct Preference Optimization22 Feb
  2. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  3. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization11 Oct
  4. 2024Learning Dynamics of LLM Finetuning15 Jul
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  7. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  8. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  10. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  11. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 45

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  11. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
33 more
  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  2. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  3. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  4. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  5. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  6. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  7. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  8. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  9. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  10. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  11. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  12. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  13. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  14. 2024Iterative Reasoning Preference Optimizationcited by 244
  15. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  16. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  17. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  18. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  19. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  20. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  21. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  22. 2022When Do Flat Minima Optimizers Work?cited by 78
  23. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  24. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  25. 2024Why is SAM Robust to Label Noise?cited by 21
  26. 2025Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimizationcited by 21
  27. 2022Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspectivecited by 15
  28. 2025CPGD: Toward Stable Rule-based Reinforcement Learning for Language Modelscited by 11
  29. 2023Flat Seeking Bayesian Neural Networkscited by 10
  30. 2025Avoiding spurious sharpness minimization broadens applicability of SAMcited by 8
  31. 2023Optimal Transport Model Distributional Robustnesscited by 5
  32. 2024Improving Generalization with Flat Hilbert Bayesian Inferencecited by 5
  33. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026TokenRatio: Principled Token-Level Preference Optimization via Ratio Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models
  2. 2025DPO-Shift: Shifting the Distribution of Direct Preference Optimization
  3. 2026Disentangling Optimization Scale from Preference Scale in DPO
  4. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
  5. 2026LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization
  6. 2025Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimization
  7. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  8. 2024Disentangling Length from Quality in Direct Preference Optimization