mlsanityData to 14 Oct 2026

Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment

arXiv 2510.05526 ↗First posted 7 Oct 2025cs.LG cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) are important techniques to align large language models (LLM) with human preference. However, the quality of RLHF and DPO training is seriously compromised by textitCorrupted preference, reward textitOveroptimization, and bias towards textitVerbosity. To our knowledge, most existing works tackle only one of these important issues, and the few other works require much computation to estimate multiple reward models and lack theoretical guarantee of generalization ability. In this work, we propose RLHF-COV and DPO-COV algorithms that can simultaneously mitigate these three issues, in both offline and online settings. This ability is theoretically demonstrated by obtaining length-regularized generalization error rates for our DPO-COV algorithms trained on corrupted data, which match the best-known rates for simpler cases with clean data and without length regularization. Moreover, our DPO-COV algorithm is simple to implement without reward estimation, and is proved to be equivalent to our RLHF-COV algorithm, which directly implies the equivalence between the vanilla RLHF and DPO algorithms. Experiments demonstrate the effectiveness of our DPO-COV algorithms under both offline and online settings.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 71 references that are in the map. First posted on arXiv 7 Oct 2025.

Builds on

56 of 71 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization18 Jul
  2. 2024Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level17 Jun
  3. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models6 Jun
  4. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment29 May
  5. 2024Robust Preference Optimization through Reward Model Distillation29 May
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  7. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  8. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  9. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  11. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 45

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  10. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  12. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
33 more
  1. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  3. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  5. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  6. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  7. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  8. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  9. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 200
  10. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
  11. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  12. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 136
  13. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  14. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  15. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  16. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  17. 2023Is RLHF More Difficult than Standard RL?cited by 96
  18. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  19. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  20. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  21. 2017Online Reinforcement Learning in Stochastic Gamescited by 67
  22. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  23. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 56
  24. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  25. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  26. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 39
  27. 2023A Theoretical Analysis of Optimistic Proximal Policy Optimization in Linear Markov Decision Processescited by 29
  28. 2024Robust Reinforcement Learning from Corrupted Human Feedbackcited by 25
  29. 2023Reinforcement Learning with Human Feedback for Realistic Traffic Simulationcited by 21
  30. 2023Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Explorationcited by 20
  31. 2024Corruption Robust Offline Reinforcement Learning with Human Feedbackcited by 20
  32. 2023Optimistic Natural Policy Gradient: a Simple Efficient Policy Optimization Framework for Online RLcited by 13
  33. 2024Reinforcement Learning from Human Feedback for Lane Changing of Autonomous Vehicles in Mixed Trafficcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models
  2. 2025A Principled Loss Function for Direct Language Model Alignment
  3. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
  4. 2024Bootstrapping Language Models with DPO Implicit Rewards
  5. 2025Online Distributionally Robust LLM Alignment via Regression to Relative Reward
  6. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model
  7. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  8. 2024Filtered Direct Preference Optimization