mlsanityData to 7 Oct 2026

The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models

arXiv 2410.06554 ↗First posted 9 Oct 2024cs.CL cs.AIcited by 19

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback significantly enhances Natural Language Processing by aligning language models with human expectations. A critical factor in this alignment is the strength of reward models used during training. This study explores whether stronger reward models invariably lead to better language models. In this paper, through experiments on relevance, factuality, and completeness tasks using the QA-FEEDBACK dataset and reward models based on Longformer, we uncover a surprising paradox: language models trained with moderately accurate reward models outperform those guided by highly accurate ones. This challenges the widely held belief that stronger reward models always lead to better language models, and opens up new avenues for future research into the key factors driving model performance and how to choose the most suitable reward models. Code and additional details are available at https://github.com/EIT-NLP/AccuracyParadox-RLHF.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 29 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

21 of 29 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs12 Apr
  2. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performance7 Apr
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  4. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  12. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
4 more
  1. 2022Reward Gaming in Conditional Text Generationcited by 29
  2. 2024Unraveling the Mystery of Scaling Laws: Part Icited by 24
  3. 2022WeLM: A Well-Read Pre-trained Language Model for Chinesecited by 19
  4. 2019Select and Attend: Towards Controllable Content Selection in Text Generationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 7.

19 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  2. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  3. 2025RewardDance: Reward Scaling in Visual Generationcited by 63
  4. 2025RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Datacited by 11
  5. 2025Doubly Robust Alignment for Large Language Modelscited by 11
  6. 2025ReDit: Reward Dithering for Improved LLM Policy Optimizationcited by 8
  7. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  8. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  9. 2025Mirage or Method? How Model-Task Alignment Induces Divergent RL Conclusionscited by 6
  10. 2026RVPO: Risk-Sensitive Alignment via Variance Regularizationcited by 3
  11. 2026Discretizing Reward Modelscited by 3
  12. 2025Accelerating RLHF Training with Reward Variance Increasecited by 2
7 more
  1. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  2. 2025Explainable reinforcement learning from human feedback to improve alignmentcited by 1
  3. 2025Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedbackcited by 0
  4. 2026Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedbackcited by 0
  5. 2026Suan: Rectifying Direct Preference Safety Alignment in Large Language Modelscited by 0
  6. 2026EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learningcited by 0
  7. 2026Revisiting On-policy Adversarial Black-Box Distillation: Calibrating Groupwise Reward Geometry for Effective Advantage Constructioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  2. 2024How to Evaluate Reward Models for RLHF
  3. 2023The Trickle-down Impact of Reward (In-)consistency on RLHF
  4. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  5. 2025A Systematic Analysis of Base Model Choice for Reward Modeling
  6. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  7. 2026When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
  8. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models