Data to 14 Oct 2026

Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control

arXiv 2405.16158 ↗First posted 25 May 2024cs.LGcited by 108

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Sample efficiency in Reinforcement Learning (RL) has traditionally been driven by algorithmic enhancements. In this work, we demonstrate that scaling can also lead to substantial improvements. We conduct a thorough investigation into the interplay of scaling model capacity and domain-specific RL enhancements. These empirical findings inform the design choices underlying our proposed BRO (Bigger, Regularized, Optimistic) algorithm. The key innovation behind BRO is that strong regularization allows for effective scaling of the critic networks, which, paired with optimistic exploration, leads to superior performance. BRO achieves state-of-the-art results, significantly outperforming the leading model-based and model-free algorithms across 40 complex tasks from the DeepMind Control, MetaWorld, and MyoSuite benchmarks. BRO is the first model-free algorithm to achieve near-optimal policies in the notoriously challenging Dog and Humanoid tasks.

Key facts. Cited by 108 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 46 of its 63 references that are in the map. First posted on arXiv 25 May 2024.

Builds on

46 of 63 references.

Closest ancestors · 4

Same direction, newest first

Other references · 42

2017Attention Is All You Needcited by 60,361
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
2016Layer Normalizationcited by 5,669
2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
2023Mastering Diverse Domains through World Modelscited by 1,253
2018Soft Actor-Critic Algorithms and Applicationscited by 1,240

30 more

2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
2018DeepMind Control Suitecited by 956
2020On Layer Normalization in the Transformer Architecturecited by 922
2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
2017A Distributional Perspective on Reinforcement Learningcited by 765
2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
2023Efficient Online Reinforcement Learning with Offline Datacited by 363
2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
2022The Primacy Bias in Deep Reinforcement Learningcited by 247
2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
2019Optimism in Reinforcement Learning with Generalized Linear Function Approximationcited by 111
2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
2019Better Exploration with Optimistic Actor-Criticcited by 88
2022MyoSuite -- A contact-rich simulation suite for musculoskeletal motor controlcited by 80
2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 73
2018Improving Regression Performance with Distributional Lossescited by 67
2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
2023Small batch deep reinforcement learningcited by 36
2018Expected Policy Gradients for Reinforcement Learningcited by 28
2015How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategiescited by 27
2023A Generalist Dynamics Model for Controlcited by 24
2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
2021Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learningcited by 18
2023On the Theory of Risk-Aware Agents: Bridging Actor-Critic and Economicscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 54; 2026: 47.

108 papers.

Sort:

By area:

2025Flow Q-Learningcited by 184
2024Simplifying Deep Temporal Difference Learningcited by 78
2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 76
2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 58
2025Horizon Reduction Makes RL Scalablecited by 57
2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 56
20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 45
2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 44
2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 38
2025Maximum Entropy Reinforcement Learning with Diffusion Policycited by 34
2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23

38 more

2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 22
2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19
2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
2025Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returnscited by 16
2025Learning Sim-to-Real Humanoid Locomotion in 15 Minutescited by 16
2025Value-Based Deep RL Scales Predictablycited by 15
2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learningcited by 11
2026EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Modelscited by 11
2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 10
2024Active Fine-Tuning of Multi-Task Policiescited by 9
2025A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Controlcited by 9
2026TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapsecited by 9
2025Multi-Task Reinforcement Learning Enables Parameter Scalingcited by 8
2025SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfercited by 8
2025Hadamax Encoding: Elevating Performance in Model-Free Ataricited by 7
2025Relative Entropy Pathwise Policy Optimizationcited by 7
2026RynnValue: Scaling Robotic Value Foundation Models with Temporal Distancecited by 7
2025Eau De Q-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learningcited by 6
2025Impoola: The Power of Average Pooling for Image-Based Deep Reinforcement Learningcited by 6
2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learningcited by 6
2025Compute-Optimal Scaling for Value-Based Deep RLcited by 6
2025BuilderBench: The Building Blocks of Intelligent Agentscited by 6
2025NIL: No-data Imitation Learning by Leveraging Pre-trained Video Diffusion Modelscited by 5
2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 5
2025Evolutionary Policy Optimizationcited by 4
2025Plasticity-Aware Mixture of Experts for Learning Under QoE Shifts in Adaptive Video Streamingcited by 4
2025Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learningcited by 4
2025Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learningcited by 4
2025Is Temporal Difference Learning the Gold Standard for Stitching in RL?cited by 4
2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulationcited by 4
2025ZeroSiam: An Efficient Asymmetry for Test-Time Entropy Optimization without Collapsecited by 3
2025Off-policy Reinforcement Learning with Model-based Exploration Augmentationcited by 3
2025Understanding and Improving Hyperbolic Deep Reinforcement Learningcited by 3

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way