mlsanityData to 14 Oct 2026

Boosting Direct Preference Optimization with Penalization

arXiv 2606.12505 ↗First posted 10 Jun 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset. This leaves a useful signal unused: the response that the reference model itself would generate for the same prompt. We propose Direct Preference Optimization with Penalization (DPOP), a simple extension of DPO that augments the base preference loss with a gated penalty on reference-greedy responses. DPOP activates this penalty only when the current policy still assigns a lower likelihood to the preferred response than to the rejected response. On AlpacaEval 2.0, DPOP improves length-controlled win rate over DPO, SimPO, and AlphaDPO on both Llama-3-8b-it and Gemma-2-9b-it, achieving relative gains of 5.3% and 4.4% over baselines on the two models, respectively. Ablations further show that a SimNPO-style length-normalized penalty is stronger than NPO and token-level unlikelihood in this setting.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 8 of its 9 references that are in the map. First posted on arXiv 10 Jun 2026.

Builds on

8 of 9 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization14 Oct
  2. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  4. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 3

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  3. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model
  2. 2025Length-Controlled Margin-Based Preference Optimization without Reference Model
  3. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
  4. 2024MallowsPO: Fine-Tune Your LLM with Preference Dispersions
  5. 2024New Desiderata for Direct Preference Optimization
  6. 2026AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
  7. 2025Active Learning for Direct Preference Optimization
  8. 2024Length Desensitization in Direct Preference Optimization