mlsanityData to 14 Oct 2026

Stabilizing RLHF through Advantage Model and Selective Rehearsal

arXiv 2309.10202 ↗First posted 18 Sep 2023cs.CL cs.AIcited by 14

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Large Language Models (LLMs) have revolutionized natural language processing, yet aligning these models with human values and preferences using RLHF remains a significant challenge. This challenge is characterized by various instabilities, such as reward hacking and catastrophic forgetting. In this technical report, we propose two innovations to stabilize RLHF training: 1) Advantage Model, which directly models advantage score i.e., extra reward compared to the expected rewards and regulates score distributions across tasks to prevent reward hacking. 2) Selective Rehearsal, which mitigates catastrophic forgetting by strategically selecting data for PPO training and knowledge rehearsing. Our experimental analysis on public and proprietary datasets reveals that the proposed methods not only increase stability in RLHF training but also achieve higher reward scores and win rates.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 35 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

30 of 35 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  6. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  12. 2023LIMA: Less Is More for Alignmentcited by 1,100
11 more
  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  4. 2023Instruction Tuning with GPT-4cited by 686
  5. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  6. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  7. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  8. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  9. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 137
  10. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasetscited by 61
  11. 2021ValueNet: A New Dataset for Human Value Driven Dialogue Systemcited by 34

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 6; 2025: 4.

14 papers.

Sort:

By area:

  1. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 78
  2. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 72
  3. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  4. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  5. 2024Modality-Fair Preference Optimization for Trustworthy MLLM Alignmentcited by 26
  6. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descentcited by 24
  7. 2024HyperCLOVA X Technical Reportcited by 18
  8. 2023Reasons to Reject? Aligning Language Models with Judgmentscited by 15
  9. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  10. 2025Mapping Post-Training Forgetting in Language Models at Scalecited by 11
  11. 2025Rethinking the Sampling Criteria in Reinforcement Learning for LLM Reasoning: A Competence-Difficulty Alignment Perspectivecited by 9
  12. 2023Enabling Language Models to Implicitly Learn Self-Improvementcited by 7
2 more
  1. 2023COPR: Continual Learning Human Preference through Optimal Policy Regularizationcited by 2
  2. 2025HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Reward-Robust RLHF in LLMs
  2. 2026Robust Policy Optimization to Prevent Catastrophic Forgetting
  3. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  4. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  5. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  6. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models
  7. 2023Mitigating the Alignment Tax of RLHF
  8. 2024The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models