mlsanityData to 14 Oct 2026

Value-Based Deep RL Scales Predictably

arXiv 2502.04327 ↗First posted 6 Feb 2025cs.LGcited by 15

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Scaling data and compute is critical to the success of modern ML. However, scaling demands predictability: we want methods to not only perform well with more compute or data, but also have their performance be predictable from small-scale runs, without running the large-scale experiment. In this paper, we show that value-based off-policy RL methods are predictable despite community lore regarding their pathological behavior. First, we show that data and compute requirements to attain a given performance level lie on a Pareto frontier, controlled by the updates-to-data (UTD) ratio. By estimating this frontier, we can predict this data requirement when given more compute, and this compute requirement when given more data. Second, we determine the optimal allocation of a total resource budget across data and compute for a given performance and use it to determine hyperparameters that maximize performance for a given budget. Third, this scaling is enabled by first estimating predictable relationships between hyperparameters, which is used to manage effects of overfitting and plasticity loss unique to RL. We validate our approach using three algorithms: SAC, BRO, and PQL on DeepMind Control, OpenAI gym, and IsaacGym, when extrapolating to higher levels of data, compute, budget, or performance.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 46 of its 57 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

46 of 57 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning13 Oct
  2. 2024Simplifying Deep Temporal Difference Learning5 Jul
  3. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control25 May

Other references · 43

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2015Continuous control with deep reinforcement learningcited by 3,407
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  11. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  12. 2023Mastering Diverse Domains through World Modelscited by 1,253
31 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  2. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  3. 2018DeepMind Control Suitecited by 956
  4. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  6. 2023Scaling Data-Constrained Language Modelscited by 403
  7. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  8. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  9. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  10. 2018An Empirical Model of Large-Batch Trainingcited by 281
  11. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  12. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  13. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  14. 2023Understanding plasticity in neural networkscited by 174
  15. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  16. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  17. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  18. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  19. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  20. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 73
  21. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  22. 2021Scaling Scaling Laws with Board Gamescited by 63
  23. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 62
  24. 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
  25. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  26. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 47
  27. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  28. 2023Scaling laws for single-agent reinforcement learningcited by 41
  29. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  30. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Modelscited by 30
  31. 2024SAPG: Split and Aggregate Policy Gradientscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 7.

15 papers.

Sort:

By area:

  1. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  2. 2025Horizon Reduction Makes RL Scalablecited by 57
  3. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  4. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 22
  5. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
  6. 2025Compute-Optimal Scaling for Value-Based Deep RLcited by 6
  7. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
  8. 2026IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RLcited by 6
  9. 2026Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environmentscited by 3
  10. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 1
  11. 2026When Does Non-Uniform Replay Matter in Reinforcement Learning?cited by 0
  12. 2026Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learningcited by 0
3 more
  1. 2026Rank Collapse Is Recoverable, Growing |Q| Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Criticcited by 0
  2. 2026Sample Complexity of Equivariant Reinforcement Learningcited by 0
  3. 2026Single or Multiple Policies for Phase-Structured Reinforcement Learning?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergence
  2. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters
  3. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
  4. 2026When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control
  5. 2025Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers
  6. 2026Model-Agnostic Solutions for Deep Reinforcement Learning in Non-Ergodic Contexts
  7. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RL
  8. 2021Instabilities of Offline RL with Pre-Trained Neural Representation