mlsanityData to 7 Oct 2026

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization

arXiv 2407.13399 ↗First posted 18 Jul 2024cs.AI cs.CL cs.LGcited by 78

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Language model alignment methods such as reinforcement learning from human feedback (RLHF) have led to impressive advances in language model capabilities, but are limited by a widely observed phenomenon known as overoptimization, where the quality of the language model degrades over the course of the alignment process. As the model optimizes performance with respect to an offline reward model, it overfits to inaccuracies and drifts away from preferred responses covered by the data. To discourage such distribution shift, KL-regularization is widely employed in existing offline alignment methods, but overoptimization continues to harm performance. Lending theoretical insight into the source of these empirical observations, we first show that the KL-regularization is too weak to prevent overfitting, then raise the following question: is it possible to design an efficient algorithm that is provably robust to overoptimization? We address this question with a new algorithm for offline alignment, chi2-Preference Optimization (chiPO). chiPO is a one-line change to Direct Preference Optimization (DPO; Rafailov et al., 2023), which only involves modifying the logarithmic link function in the DPO objective. Despite this minimal change, chiPO implicitly implements the principle of pessimism in the face of uncertainty via regularization with the chi2-divergence -- which quantifies uncertainty more effectively than KL-regularization -- and provably alleviates overoptimization, achieving sample-complexity guarantees based on single-policy concentrability -- the gold standard in offline reinforcement learning. chiPO's simplicity and strong guarantees make it the first practical and general-purpose offline alignment algorithm that is provably robust to overoptimization.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 103 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

48 of 103 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models6 Jun
  2. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coverage3 Jun
  3. 2024Robust Preference Optimization through Reward Model Distillation29 May
  4. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  5. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  6. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  7. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  8. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  9. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2023PaLM 2 Technical Reportcited by 1,154
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  6. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  8. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  9. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  10. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  11. 2023Nash Learning from Human Feedbackcited by 220
  12. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
27 more
  1. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  2. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  3. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  4. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  5. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  6. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  7. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  8. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  9. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  10. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  11. 2023Is RLHF More Difficult than Standard RL?cited by 96
  12. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  13. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  14. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
  15. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  16. 2024Active Preference Optimization for Sample Efficient RLHFcited by 61
  17. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  18. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  19. 2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
  20. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  21. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  22. 2020Understanding Learned Reward Functionscited by 37
  23. 2023Foundations of Reinforcement Learning and Interactive Decision Makingcited by 33
  24. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
  25. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learningcited by 25
  26. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilizationcited by 25
  27. 2024Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learningcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 41; 2026: 31.

78 papers.

Sort:

By area:

  1. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  2. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  3. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  4. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  5. 2025A Survey of Direct Preference Optimizationcited by 35
  6. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  7. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  8. 2025On a few pitfalls in KL divergence gradient estimation for RLcited by 27
  9. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 23
  10. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 20
  11. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?cited by 20
  12. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 17
38 more
  1. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 15
  2. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 15
  3. 2025SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewardscited by 14
  4. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  5. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexitycited by 11
  6. 2026Future-KL Regularized GRPO: Process-Level Credit Assignment from f-Divergence Regularizationcited by 11
  7. 2024Forward KL Regularized Preference Optimization for Aligning Diffusion Policiescited by 10
  8. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
  9. 2024Faster WIND: Accelerating Iterative Best-of-N Distillation for LLM Alignmentcited by 9
  10. 2025Towards a Sharp Analysis of Offline Policy Learning for f-Divergence-Regularized Contextual Banditscited by 9
  11. 2025Preference Optimization by Estimating the Ratio of the Data Distributioncited by 9
  12. 2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPOcited by 8
  13. 2025Mitigating Preference Hacking in Policy Optimization with Pessimismcited by 7
  14. 2025Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Modelscited by 6
  15. 2025RSPO: Regularized Self-Play Alignment of Large Language Modelscited by 5
  16. 2025Learning a Pessimistic Reward Model in RLHFcited by 5
  17. 2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hackingcited by 5
  18. 2026Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferencescited by 5
  19. 2025Design Considerations in Offline Preference-based RLcited by 4
  20. 2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspectivecited by 4
  21. 2025RePO: Understanding Preference Learning Through ReLU-Based Optimizationcited by 4
  22. 2025PPO in the Fisher-Rao geometrycited by 4
  23. 2025Preference learning made easy: Everything should be understood through win ratecited by 3
  24. 2025SquarechiPO: Differentially Private and Robust chi2-Preference Optimization in Offline Direct Alignmentcited by 3
  25. 2025Beyond RLHF: A Unified Theoretical Framework of Alignmentcited by 3
  26. 2025Offline and Online KL-Regularized RLHF under Differential Privacycited by 3
  27. 2025Improved Bounds for Private and Robust Alignmentcited by 3
  28. 2026f-Trajectory Balance: A Loss Family for Tuning GFlowNets, Generative Models, and LLMs with Off- and On-Policy Datacited by 3
  29. 2024Meta-Learning Objectives for Preference Optimizationcited by 2
  30. 2025C2-DPO: Constrained Controlled Direct Preference Optimizationcited by 2
  31. 2025Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimizationcited by 2
  32. 2025Non-Asymptotic Global Convergence of PPO-Clipcited by 2
  33. 2025Semiparametric Preference Optimization: Your Language Model is Secretly a Single-Index Modelcited by 2
  34. 2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gainscited by 2
  35. 2026Semantic-aware Wasserstein Policy Regularization for Large Language Model Alignmentcited by 2
  36. 2026Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignmentcited by 2
  37. 2026Robust Optimization for Mitigating Reward Hacking with Correlated Proxiescited by 2
  38. 2026f-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analysescited by 2

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Theoretical Limits of Language Model Alignment
  2. 2024Decoding-time Realignment of Language Models
  3. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  4. 2025Doubly Robust Alignment for Large Language Models
  5. 2024Towards Efficient Exact Optimization of Language Model Alignment
  6. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
  7. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
  8. 2024Asymptotics of Language Model Alignment