mlsanityData to 14 Oct 2026

Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling

arXiv 2506.08681 ↗First posted 10 Jun 2025cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Alignment Algorithms (DAAs) such as Direct Preference Optimization (DPO) have emerged as alternatives to the standard Reinforcement Learning from Human Feedback (RLHF) for aligning large language models (LLMs) with human values. However, these methods are more susceptible to over-optimization, in which the model drifts away from the reference policy, leading to degraded performance as training progresses. This paper proposes a novel importance-sampling approach to mitigate the over-optimization problem of offline DAAs. This approach, called (IS-DAAs), multiplies the DAA objective with an importance ratio that accounts for the reference policy distribution. IS-DAAs additionally avoid the high variance issue associated with importance sampling by clipping the importance ratio to a maximum value. Our extensive experiments demonstrate that IS-DAAs can effectively mitigate over-optimization, especially under low regularization strength, and achieve better performance than other methods designed to address this problem. Our implementations are provided publicly at this link.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 37 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

30 of 37 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization18 Jul
  2. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models21 Jun
  3. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coverage3 Jun
  4. 2024Preference Learning Algorithms Do Not Learn Preference Rankings29 May
  5. 2024Robust Preference Optimization through Reward Model Distillation29 May
  6. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  7. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  8. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  9. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  10. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  11. 2024Towards Efficient Exact Optimization of Language Model Alignment1 Feb
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
1 more
  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 17

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  10. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  11. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 136
  12. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
5 more
  1. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  2. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  3. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 69
  4. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  5. 2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33

Similar, not cited

Close in wording, no citation either way

  1. 2024Understanding Likelihood Over-optimisation in Direct Alignment Algorithms
  2. 2025Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms
  3. 2024SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins
  4. 2026Normalized Rewards for Preference Optimization
  5. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights
  6. 2025AlphaPO: Reward Shape Matters for LLM Alignment
  7. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models
  8. 2024As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss