mlsanityData to 7 Oct 2026

Does RLHF Scale? Exploring the Impacts From Data, Model, and Method

arXiv 2412.06000 ↗First posted 8 Dec 2024cs.CL cs.LGcited by 22

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

This study explores the scaling properties of Reinforcement Learning from Human Feedback (RLHF) in Large Language Models (LLMs). Although RLHF is considered an important step in post-training of LLMs, its scaling potential is still largely unknown. We systematically analyze key components in the RLHF framework--model size, data composition, and inference budget--and their impacts on performance. Our findings show that increasing data diversity and volume improves reward model performance, helping process-supervision models scale better. For policy training, more response samples per prompt boost performance initially but quickly plateau. And larger reward models offer modest gains in policy training. In addition, larger policy models benefit less from RLHF with a fixed reward model. Overall, RLHF scales less efficiently than pretraining, with diminishing returns from additional computational resources. Based on these observations, we propose strategies to optimize RLHF performance within computational limits.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 41 references that are in the map. First posted on arXiv 8 Dec 2024.

Builds on

33 of 41 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms5 Jun
  2. 2024OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework20 May
  3. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  4. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
14 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  6. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  7. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  8. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  9. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  10. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  11. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  12. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  13. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  14. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 5.

22 papers.

Sort:

By area:

  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  2. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  3. 2025Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solvingcited by 62
  4. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  5. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 27
  6. 2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedbackcited by 26
  7. 2025SWE-Dev: Building Software Engineering Agents with Training and Inference Scalingcited by 24
  8. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  9. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  10. 2025R2Vul: Learning to Reason about Software Vulnerabilities with Reinforcement Learning and Structured Reasoning Distillationcited by 7
  11. 2025Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learningcited by 4
  12. 2025VARD: Efficient and Dense Fine-Tuning for Diffusion Models with Value-based RLcited by 3
10 more
  1. 2025Value Under Ignorance in Universal Artificial Intelligencecited by 3
  2. 2025Retrieval-Augmented Fine-Tuning With Preference Optimization For Visual Program Generationcited by 2
  3. 2025RETuning: Upgrading Inference-Time Scaling for Stock Movement Prediction with Large Language Modelscited by 1
  4. 2026TARo: Token-level Adaptive Routing for LLM Test-time Alignmentcited by 1
  5. 2025Tiny Reward Modelscited by 0
  6. 2025On The Finetuning of MLIPs Through the Lens of Iterated Maps With BPTTcited by 0
  7. 2026Addressing Performance Saturation for LLM RL via Precise Entropy Curve Controlcited by 0
  8. 2026SALT: When More Rollouts Don't Help in Group-Based Policy Optimization and How to Make Them Mattercited by 0
  9. 2026Performance Foundations of Parallel & Distributed Reasoning Language Modelscited by 0
  10. 2026Scaling of Capability and Efficiency at Inference Time in Large Reasoning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  2. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  3. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversity
  4. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization
  5. 2024How to Evaluate Reward Models for RLHF
  6. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  8. 2024RLHF Workflow: From Reward Modeling to Online RLHF