mlsanityData to 14 Oct 2026

Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback

arXiv 2503.22230 ↗First posted 28 Mar 2025cs.LGcited by 26

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning large language models with human preferences. While recent research has focused on algorithmic improvements, the importance of prompt-data construction has been overlooked. This paper addresses this gap by exploring data-driven bottlenecks in RLHF performance scaling, particularly reward hacking and decreasing response diversity. We introduce a hybrid reward system combining reasoning task verifiers (RTV) and a generative reward model (GenRM) to mitigate reward hacking. We also propose a novel prompt-selection method, Pre-PPO, to maintain response diversity and enhance learning effectiveness. Additionally, we find that prioritizing mathematical and coding tasks early in RLHF training significantly improves performance. Experiments across two model sizes validate our methods' effectiveness and scalability. Results show that RTV is most resistant to reward hacking, followed by GenRM with ground truth, and then GenRM with SFT Best-of-N responses. Our strategies enable rapid capture of subtle task-specific distinctions, leading to substantial improvements in overall RLHF performance. This work highlights the importance of careful data construction and provides practical methods to overcome performance barriers in RLHF.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 40 references that are in the map. First posted on arXiv 28 Mar 2025.

Builds on

26 of 40 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Method8 Dec
  2. 2024Generative Reward Models2 Oct
  3. 2024BOND: Aligning LLMs with Best-of-N Distillation19 Jul
  4. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  5. 2023The History and Risks of Reinforcement Learning and Human Feedback20 Oct
  6. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversity10 Oct
  7. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2023On The Fragility of Learned Reward Functions9 Jan
  10. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 15

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  8. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  9. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
  10. 2023Reward Design with Language Modelscited by 261
  11. 2025LIMR: Less is More for RL Scalingcited by 139
  12. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
3 more
  1. 2023Calibrated Language Models Must Hallucinatecited by 115
  2. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  3. 2021Improved Fine-Tuning by Better Leveraging Pre-Training Datacited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 6.

26 papers.

Sort:

By area:

  1. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  2. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 161
  3. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 93
  4. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  5. 2025AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learningcited by 87
  6. 2025LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMscited by 70
  7. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzlescited by 56
  8. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 54
  9. 2025TreeRPO: Tree Relative Policy Optimizationcited by 46
  10. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 42
  11. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 33
  12. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
14 more
  1. 2026Real-Time Aligned Reward Model beyond Semanticscited by 13
  2. 2025MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Clustercited by 9
  3. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Modelcited by 6
  4. 2025Enhancing Large Language Models through Structured Reasoningcited by 5
  5. 2025DRIVE: Data Curation Best Practices for Reinforcement Learning with Verifiable Reward in Competitive Code Generationcited by 3
  6. 2026When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropycited by 3
  7. 2025G-UBS: Towards Robust Understanding of Implicit Feedback via Group-Aware User Behavior Simulationcited by 2
  8. 2026Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Modelscited by 1
  9. 2025Balancing Robustness and Efficiency in Embedded DNNs Through Activation Function Selectioncited by 0
  10. 2025Break Stylistic Sophon: Are We Really Meant to Confine the Imagination in Style Transfer?cited by 0
  11. 2025A First-Order Logic-Based Alternative to Reward Models in RLHFcited by 0
  12. 2026Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimescited by 0
  13. 2026Imitation Learning for Elder-Facing Speech Synthesiscited by 0
  14. 2026Scaling Properties of Same-Family On-Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  3. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  4. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  5. 2026RMB: Reward Model Boosting Mitigates Reward Hacking
  6. 2025Provably Efficient Online RLHF with One-Pass Reward Modeling
  7. 2024Prototypical Reward Network for Data-Efficient RLHF
  8. 2025Curiosity-Driven Reinforcement Learning from Human Feedback