mlsanityData to 7 Oct 2026

SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment

arXiv 2505.12435 ↗First posted 18 May 2025cs.LG cs.AI cs.DCcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) is broadly utilized for aligning Large Language Models (LLMs) with human values because of its flexibility. Despite its effectiveness, it has been observed that the capability of DPO to generate human-preferred response is limited and the results of DPO are far from resilient. To address these limitations, in this paper we propose a novel Self-Guided Direct Preference Optimization algorithm, i.e., SGDPO, which incorporates a pilot term to steer the gradient flow during the optimization process, allowing for fine-grained control over the updates of chosen and rejected rewards. We provide a detailed theoretical analysis of our proposed method and elucidate its operational mechanism. Furthermore, we conduct comprehensive experiments on various models and benchmarks. The extensive experimental results demonstrate the consistency between the empirical results and our theoretical analysis and confirm the effectiveness of our proposed approach (up to 9.19% higher score).

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 56 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

41 of 56 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters2 Feb
  2. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  3. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization11 Oct
  4. 2024Length Desensitization in Direct Preference Optimization10 Sep
  5. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence16 Jun
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  7. 2024Binary Classifier Optimization for Large Language Model Alignment6 Apr
  8. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective6 Apr
  9. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback1 Apr
  10. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  11. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  12. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
4 more
  1. 2024Noise Contrastive Alignment of Language Models with Explicit Rewards8 Feb
  2. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 25

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  11. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  12. 2025A Comprehensive Survey on Long Context Language Modelingcited by 113
13 more
  1. 2021From Distributed Machine Learning to Federated Learning: A Surveycited by 65
  2. 2023Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimizationcited by 65
  3. 2023FedASMU: Efficient Asynchronous Federated Learning with Dynamic Staleness-aware Model Updatecited by 32
  4. 2021Efficient Device Scheduling with Multi-Job Federated Learningcited by 29
  5. 2022FedDUAP: Federated Learning with Dynamic Update and Adaptive Pruning Using Shared Data on the Servercited by 29
  6. 2022Multi-Job Intelligent Scheduling with Cross-Device Federated Learningcited by 20
  7. 2021HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environmentscited by 15
  8. 2023AEDFL: Efficient Asynchronous Decentralized Federated Learning with Heterogeneous Devicescited by 15
  9. 2024Trustworthy Federated Learning: Privacy, Security, and Beyondcited by 11
  10. 2023Distributed and Deep Vertical Federated Learning with Big Datacited by 10
  11. 2023Efficient Asynchronous Federated Learning with Sparsification and Quantizationcited by 9
  12. 2024Efficient Federated Learning Using Dynamic Update and Adaptive Pruning with Momentum on Shared Server Datacited by 6
  13. 2025Addressing Cold-start Problem in Click-Through Rate Prediction via Supervised Diffusion Modelingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026SCMAPR: Self-Correcting Multi-Agent Prompt Refinement for Complex-Scenario Text-to-Video Generationcited by 4
  2. 2026AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updatescited by 1
  3. 2026TSPORec: Token Selection via Preference Optimization for LLM-Based Sequential Recommendationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Direct Preference Optimization with an Offset
  2. 2025SGPO: Self-Generated Preference Optimization based on Self-Improver
  3. 2024Towards Efficient Exact Optimization of Language Model Alignment
  4. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization
  5. 2024Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment
  6. 2024sDPO: Don't Use Your Data All at Once
  7. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  8. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective