mlsanityData to 7 Oct 2026

It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF

arXiv 2406.07971 ↗First posted 12 Jun 2024cs.CL cs.AI cs.LGcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) involves training policy models (PMs) and reward models (RMs) to align language models with human preferences. Instead of focusing solely on PMs and RMs independently, we propose to examine their interactions during fine-tuning, introducing the concept of seamlessness. Our study starts with observing the saturation phenomenon, where continual improvements in RM and PM do not translate into RLHF progress. Our analysis shows that RMs fail to assign proper scores to PM responses, resulting in a 35% mismatch rate with human preferences, highlighting a significant discrepancy between PM and RM. To measure seamlessness between PM and RM without human effort, we propose an automatic metric, SEAM. SEAM quantifies the discrepancies between PM and RM judgments induced by data samples. We validate the effectiveness of SEAM in data selection and model augmentation. Our experiments demonstrate that (1) using SEAM-filtered data for RL training improves RLHF performance by 4.5%, and (2) SEAM-guided model augmentation results in a 4% performance improvement over standard augmentation methods.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 49 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

42 of 49 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback31 Oct
  2. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  3. 2023The Trickle-down Impact of Reward (In-)consistency on RLHF28 Sep
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  6. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  7. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  8. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  9. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 33

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  9. 2023LIMA: Less Is More for Alignmentcited by 1,080
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  11. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  12. 2023Whose Opinions Do Language Models Reflect?cited by 759
21 more
  1. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  2. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  3. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  4. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  5. 2023Are aligned neural networks adversarially aligned?cited by 305
  6. 2022Teaching language models to support answers with verified quotescited by 290
  7. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  8. 2023Pretraining Language Models with Human Preferencescited by 245
  9. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  10. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  11. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  12. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  13. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  14. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 71
  15. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 71
  16. 2017Paraphrase Generation with Deep Reinforcement Learningcited by 58
  17. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  18. 2018Toward Diverse Text Generation with Inverse Reinforcement Learningcited by 41
  19. 2018Deep Reinforcement Learning For Sequence to Sequence Modelscited by 39
  20. 2019Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluatorscited by 34
  21. 2016Bandit Structured Prediction for Learning from Partial Feedback in Statistical Machine Translationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2024Sycophancy in Large Language Models: Causes and Mitigationscited by 97
  3. 2025Visualising Policy-Reward Interplay to Inform Zeroth-Order Preference Optimisation of Large Language Modelscited by 1
  4. 2025Learning More with Less: A Dynamic Dual-Level Down-Sampling Framework for Efficient Policy Optimizationcited by 1
  5. 2026Addressing Performance Saturation for LLM RL via Precise Entropy Curve Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  3. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  4. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Method
  5. 2024How to Evaluate Reward Models for RLHF
  6. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  7. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  8. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences