mlsanityData to 7 Oct 2026

Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning

arXiv 2404.19409 ↗First posted 30 Apr 2024cs.CLcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

While Reinforcement Learning (RL) has been proven essential for tuning large language models (LLMs), it can lead to reward over-optimization (ROO). Existing approaches address ROO by adding KL regularization, requiring computationally expensive hyperparameter tuning. Additionally, KL regularization focuses solely on regularizing the language policy, neglecting a potential source of regularization: the reward function itself. Inspired by demonstration-guided RL, we here introduce the Reward Calibration from Demonstration (RCfD), which leverages human demonstrations and a reward model to recalibrate the reward objective. Formally, given a prompt, the RCfD objective minimizes the distance between the demonstrations' and LLM's rewards rather than directly maximizing the reward function. This objective shift avoids incentivizing the LLM to exploit the reward model and promotes more natural and diverse language generation. We show the effectiveness of RCfD on three language tasks, which achieves comparable performance to carefully tuned baselines while mitigating ROO.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 57 references that are in the map. First posted on arXiv 30 Apr 2024.

Builds on

40 of 57 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  2. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  4. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards7 Jun
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  8. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  9. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  9. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  10. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  11. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  12. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
18 more
  1. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  2. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  3. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  4. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  5. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  6. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
  7. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  8. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  9. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  10. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 73
  11. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  12. 2020Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learningcited by 65
  13. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 63
  14. 2020Countering Language Drift with Seeded Iterated Learningcited by 57
  15. 2019Countering Language Drift via Visual Groundingcited by 55
  16. 2023Language Model Alignment with Elastic Resetcited by 34
  17. 2019Training language GANs from Scratchcited by 31
  18. 2017End-to-end optimization of goal-driven and visually grounded dialogue systemscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 4; 2026: 5.

20 papers.

Sort:

By area:

  1. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  2. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  3. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  4. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  5. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  6. 2024Understanding the Dark Side of LLMs' Intrinsic Self-Correctioncited by 42
  7. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  8. 2024Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editingcited by 23
  9. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  10. 2024LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Banditscited by 7
  11. 2026PABU: Progress-Aware Belief Update for Efficient LLM Agentscited by 6
  12. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPOcited by 5
8 more
  1. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  2. 2026Pause or Fabricate? Training Language Models for Grounded Reasoningcited by 2
  3. 2024When Can Proxies Improve the Sample Complexity of Preference Learning?cited by 1
  4. 2025LLM-Driven Self-Refinement for Embodied Drone Task Planningcited by 1
  5. 2025MEMETRON: Metaheuristic Mechanisms for Test-time Response Optimization of Large Language Modelscited by 0
  6. 2025EAPO: Enhancing Policy Optimization with On-Demand Expert Assistancecited by 0
  7. 2026Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedbackcited by 0
  8. 2026Efficient Exploration for Iterative Nash Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs
  2. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization
  3. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
  4. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
  5. 2025Generalist Reward Models: Found Inside Large Language Models
  6. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning
  7. 2024T-REG: Preference Optimization with Token-Level Reward Regularization
  8. 2025LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs