mlsanityData to 7 Oct 2026

Self-Improving Robust Preference Optimization

arXiv 2406.01660 ↗First posted 3 Jun 2024cs.LG cs.AI stat.MLcited by 19

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Online and offline RLHF methods, such as PPO and DPO, have been highly successful in aligning AI with human preferences. Despite their success, however, these methods suffer from fundamental limitations: (a) Models trained with RLHF can learn from mistakes or negative examples through RL mechanism or contrastive loss during training. However, at inference time, they lack an innate self-improvement mechanism for error corrections. (b) The optimal solution of existing methods is highly task-dependent, making it difficult for them to generalize to new tasks. To address these challenges, we propose Self-Improving Robust Preference Optimization (SRPO), a practical and mathematically principled offline RLHF framework. The key idea behind SRPO is to cast the problem of learning from human preferences as a self-improvement process, mathematically formulated as a min-max objective that jointly optimizes a self-improvement policy and a generative policy in an adversarial fashion. Crucially, the solution for this optimization problem is independent of the training task, which makes it robust to its changes. We then show that this objective can be reformulated as a non-adversarial offline loss, which can be efficiently optimized using standard supervised learning techniques at scale. To demonstrate SRPO's effectiveness, we evaluate it using AI Win-Rate (WR) against human (GOLD) completions. When tested on the XSum dataset, SRPO outperforms DPO by a margin of 15% after 5 self revisions, achieving an impressive 90% WR. Moreover, on the challenging Arena-Hard prompts, SRPO outperforms both DPO and IPO (by 4% without revision and 6% after a single revision), reaching a 56% WR against against Llama-3.1-8B-Instruct.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 49 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

32 of 49 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  3. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  4. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  6. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  11. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  12. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
13 more
  1. 2024Self-Rewarding Language Modelscited by 654
  2. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  3. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  4. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  5. 2024Iterative Reasoning Preference Optimizationcited by 244
  6. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  7. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  8. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  9. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
  10. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  11. 2023Policy Optimization in RLHF: The Impact of Out-of-preference Datacited by 35
  12. 2019Rethinking Exposure Bias In Language Modelingcited by 9
  13. 2024Soften to Defend: Towards Adversarial Robustness via Self-Guided Label Refinementcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 3.

19 papers.

Sort:

By area:

  1. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  2. 2025A Survey of Direct Preference Optimizationcited by 35
  3. 2026UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervisioncited by 23
  4. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuitioncited by 22
  5. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  6. 2025SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Modelscited by 19
  7. 2025Evolving LLMs' Self-Refinement Capability via Synergistic Training-Inference Optimizationcited by 16
  8. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  9. 2025Sherlock: Self-Correcting Reasoning in Vision-Language Modelscited by 11
  10. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gapcited by 6
  11. 2025Towards Reason-Informed Video Editing in Unified Models with Self-Reflective Learningcited by 6
  12. 2025Safety Alignment of LMs via Non-cooperative Gamescited by 6
7 more
  1. 2026One Size, Many Fits: Aligning Diverse Group-Wise Click Preferences in Large-Scale Advertising Image Generationcited by 4
  2. 2025Self-Improving Model Steeringcited by 3
  3. 2025From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMscited by 1
  4. 2025Improving Large Vision and Language Models by Learning from a Panel of Peerscited by 1
  5. 2025Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvementcited by 0
  6. 2025DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)cited by 0
  7. 2026When Good Verifiers Go Bad: Silent Negative Transfer in Verifier-Guided VLM Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  2. 2026Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
  3. 2024Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization
  4. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  5. 2023Preference as Reward, Maximum Preference Optimization with Importance Sampling
  6. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  7. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  8. 2024New Desiderata for Direct Preference Optimization