mlsanityData to 7 Oct 2026

Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF

arXiv 2401.16335 ↗First posted 29 Jan 2024cs.LG cs.AI cs.CLcited by 57

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is a pivotal technique that aligns language models closely with human-centric values. The initial phase of RLHF involves learning human values using a reward model from ranking data. It is observed that the performance of the reward model degrades after one epoch of training, and optimizing too much against the learned reward model eventually hinders the true objective. This paper delves into these issues, leveraging the theoretical insights to design improved reward learning algorithm termed 'Iterative Data Smoothing' (IDS). The core idea is that during each training epoch, we not only update the model with the data, but also update the date using the model, replacing hard labels with soft labels. Our empirical findings highlight the superior performance of this approach over the traditional methods.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 60 of its 78 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

60 of 78 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment30 Sep
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023Reward Collapse in Aligning Large Language Models28 May
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  5. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  6. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  7. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  9. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  10. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  11. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  10. 2022Red Teaming Language Models with Language Modelscited by 983
  11. 2019Relational Knowledge Distillationcited by 873
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
37 more
  1. 2019Contrastive Representation Distillationcited by 734
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  3. 2018Born Again Neural Networkscited by 539
  4. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  5. 2019Similarity-Preserving Knowledge Distillationcited by 507
  6. 2022Decoupled Knowledge Distillationcited by 380
  7. 2019On the Efficacy of Knowledge Distillationcited by 377
  8. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  9. 2022Teaching language models to support answers with verified quotescited by 290
  10. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  11. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  12. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  13. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  14. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  15. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  16. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  17. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  18. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  19. 2020Improved Optimistic Algorithms for Logistic Banditscited by 117
  20. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  21. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  22. 2020Explaining Knowledge Distillation by Quantifying the Knowledgecited by 65
  23. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  24. 2014Reducing Dueling Bandits to Cardinal Banditscited by 62
  25. 2015Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependencecited by 59
  26. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  27. 2015Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problemcited by 51
  28. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
  29. 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
  30. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  31. 2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31
  32. 2016A Relative Exponential Weighing Algorithm for Adversarial Utility-based Dueling Banditscited by 31
  33. 2018PAC Battling Bandits in the Plackett-Luce Modelcited by 27
  34. 2018Active Ranking with Subset-wise Preferencescited by 21
  35. 2016Learning Dynamic Robot-to-Human Object Handover from Human Feedbackcited by 19
  36. 2022Exploiting Correlation to Achieve Faster Learning Rates in Low-Rank Preference Banditscited by 11
  37. 2023Towards the Fundamental Limits of Knowledge Transfer over Finite Domainscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 28; 2026: 8.

57 papers.

Sort:

By area:

  1. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  2. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  3. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  4. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  5. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
  6. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  7. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  8. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 76
  9. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  10. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  11. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  12. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
38 more
  1. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  2. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  3. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  4. 2024Optimal Design for Reward Modeling in RLHFcited by 23
  5. 2025How to Train Your Advisor: Steering Black-Box LLMs with Advisor Modelscited by 20
  6. 2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hackingcited by 18
  7. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 17
  8. 2025HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Taskscited by 16
  9. 2024Understanding the Gains from Repeated Self-Distillationcited by 14
  10. 2024SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimizationcited by 14
  11. 2024Reward Learning From Preference With Tiescited by 13
  12. 2025Collab: Controlled Decoding using Mixture of Agents for LLM Alignmentcited by 13
  13. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  14. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 12
  15. 2025Reward Model Overoptimisation in Iterated RLHFcited by 10
  16. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  17. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 10
  18. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularizationcited by 10
  19. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedbackcited by 9
  20. 2025In-context Ranking Preference Optimizationcited by 9
  21. 2025RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewardscited by 7
  22. 2025Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidancecited by 7
  23. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regretcited by 6
  24. 2024UNA: A Unified Supervised Framework for Efficient LLM Alignment Across Feedback Typescited by 6
  25. 2024Understanding Likelihood Over-optimisation in Direct Alignment Algorithmscited by 6
  26. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Explorationcited by 6
  27. 2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hackingcited by 5
  28. 2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 5
  29. 2025Energy-Based Reward Models for Robust Language Model Alignmentcited by 4
  30. 2025Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Functioncited by 4
  31. 2026Detecting and Suppressing Reward Hacking with Gradient Fingerprintscited by 4
  32. 2025Preference learning made easy: Everything should be understood through win ratecited by 3
  33. 2025C2-DPO: Constrained Controlled Direct Preference Optimizationcited by 2
  34. 2024When Can Proxies Improve the Sample Complexity of Preference Learning?cited by 1
  35. 2025The Limits of Preference Data for Post-Trainingcited by 1
  36. 2025HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Modelscited by 1
  37. 2025Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Samplingcited by 1
  38. 2025Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignmentcited by 1

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  3. 2025Provably Efficient Online RLHF with One-Pass Reward Modeling
  4. 2023SuperHF: Supervised Iterative Learning from Human Feedback
  5. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  6. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization
  7. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  8. 2026RMB: Reward Model Boosting Mitigates Reward Hacking