Linear Preference Optimization: Decoupled Gradient Control via Absolute Regularization
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
DPO (Direct Preference Optimization) has become a widely used offline preference optimization algorithm due to its simplicity and training stability. However, DPO is prone to overfitting and collapse. To address these challenges, we propose Linear Preference Optimization (LPO), a novel alignment framework featuring three key innovations. First, we introduce gradient decoupling by replacing the log-sigmoid function with an absolute difference loss, thereby isolating the optimization dynamics. Second, we improve stability through an offset constraint combined with a positive regularization term to preserve the chosen response quality. Third, we implement controllable rejection suppression using gradient separation with straightforward estimation and a tunable coefficient that linearly regulates the descent of the rejection probability. Through extensive experiments, we demonstrate that LPO consistently improves performance on various tasks, including general text tasks, math tasks, and text-to-speech (TTS) tasks. These results establish LPO as a robust and tunable paradigm for preference alignment, and we release the source code, models, and training data publicly.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 25 references that are in the map. First posted on arXiv 20 Aug 2025.
Builds on
19 of 25 references.
Closest ancestors · 7
Same direction, newest first
- 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
- 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective6 Apr
- 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
- 2024Direct Preference Optimization with an Offset16 Feb
- 2023AlignBench: Benchmarking Chinese Alignment of Large Language Models30 Nov
- 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
- 2022Training language models to follow instructions with human feedback4 Mar
Other references · 12
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2025Qwen3 Technical Reportcited by 8,820
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
- 2024Qwen2 Technical Reportcited by 2,120
- 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
- 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
- 2021Data Augmentation Approaches in Natural Language Processing: A Surveycited by 82
- 2025Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Modelscited by 50
- 2025UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic informationcited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization
- 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts
- 2024Minor DPO reject penalty to increase training robustness
- 2025Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization
- 2026AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
- 2026GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
- 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
- 2025Length-Controlled Margin-Based Preference Optimization without Reference Model