mlsanityData to 7 Oct 2026

SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning

arXiv 2410.09754 ↗First posted 13 Oct 2024cs.LG cs.AIcited by 70

Not on the map. Why?

Abstract (arXiv)

Recent advances in CV and NLP have been largely driven by scaling up the number of network parameters, despite traditional theories suggesting that larger networks are prone to overfitting. These large networks avoid overfitting by integrating components that induce a simplicity bias, guiding models toward simple and generalizable solutions. However, in deep RL, designing and scaling up networks have been less explored. Motivated by this opportunity, we present SimBa, an architecture designed to scale up parameters in deep RL by injecting a simplicity bias. SimBa consists of three components: (i) an observation normalization layer that standardizes inputs with running statistics, (ii) a residual feedforward block to provide a linear pathway from the input to output, and (iii) a layer normalization to control feature magnitudes. By scaling up parameters with SimBa, the sample efficiency of various deep RL algorithms-including off-policy, on-policy, and unsupervised methods-is consistently improved. Moreover, solely by integrating SimBa architecture into SAC, it matches or surpasses state-of-the-art deep RL methods with high computational efficiency across DMC, MyoSuite, and HumanoidBench. These results demonstrate SimBa's broad applicability and effectiveness across diverse RL algorithms and environments.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 59 of its 68 references that are in the map. First posted on arXiv 13 Oct 2024.

Builds on

59 of 68 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2016Layer Normalizationcited by 5,579
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2015Continuous control with deep reinforcement learningcited by 3,279
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  10. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  11. 2019Root Mean Square Layer Normalizationcited by 1,425
  12. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
38 more
  1. 2023Mastering Diverse Domains through World Modelscited by 1,193
  2. 2018DeepMind Control Suitecited by 935
  3. 2020On Layer Normalization in the Transformer Architecturecited by 906
  4. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  6. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  7. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  8. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  10. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  11. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  12. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  13. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  14. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  15. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  16. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  17. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  18. 2020The NetHack Learning Environmentcited by 196
  19. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  20. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  21. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  22. 2023Understanding plasticity in neural networkscited by 170
  23. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  24. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  25. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  26. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  27. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  28. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  29. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 101
  30. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 89
  31. 2022MyoSuite -- A contact-rich simulation suite for musculoskeletal motor controlcited by 80
  32. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  33. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 80
  34. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  35. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  36. 2024Simplifying Deep Temporal Difference Learningcited by 71
  37. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  38. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 35; 2026: 35.

70 papers.

Sort:

By area:

  1. 2025Flow Q-Learningcited by 162
  2. 2025Horizon Reduction Makes RL Scalablecited by 55
  3. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 39
  4. 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 32
  5. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  6. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 22
  7. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  8. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19
  9. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
  10. 2025Value-Based Deep RL Scales Predictablycited by 14
  11. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
  12. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
38 more
  1. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learningcited by 11
  2. 2025Learning Sim-to-Real Humanoid Locomotion in 15 Minutescited by 11
  3. 2025A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Controlcited by 9
  4. 2025Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoningcited by 9
  5. 2026FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoffcited by 7
  6. 2025Impoola: The Power of Average Pooling for Image-Based Deep Reinforcement Learningcited by 6
  7. 2025Hadamax Encoding: Elevating Performance in Model-Free Ataricited by 6
  8. 2025Relative Entropy Pathwise Policy Optimizationcited by 6
  9. 2025Compute-Optimal Scaling for Value-Based Deep RLcited by 5
  10. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  11. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
  12. 2025Evolutionary Policy Optimizationcited by 3
  13. 2025Off-policy Reinforcement Learning with Model-based Exploration Augmentationcited by 3
  14. 2025Spectral Representation-based Reinforcement Learningcited by 3
  15. 2026Stagewise Reinforcement Learning and the Geometry of the Regret Landscapecited by 3
  16. 2026What Makes Value Learning Efficient in Residual Reinforcement Learning?cited by 3
  17. 2026WIMLE: Uncertainty-Aware World Models with IMLE for Sample-Efficient Continuous Controlcited by 3
  18. 2026Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environmentscited by 3
  19. 2025Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learningcited by 2
  20. 2025Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learningcited by 2
  21. 2025Sparsity-Driven Plasticity in Multi-Task Reinforcement Learningcited by 2
  22. 2025Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learningcited by 2
  23. 2026Temporal Difference Learning with Constrained Initial Representationscited by 2
  24. 2026Squint: Fast Visual Reinforcement Learning for Sim-to-Real Roboticscited by 2
  25. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policiescited by 2
  26. 2026The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learningcited by 2
  27. 2026Survival Reinforcement Learning: Toward Scalable Self-Supervised RLcited by 2
  28. 2026Performance Variation in Deep Reinforcement Learningcited by 2
  29. 2025Massively Scaling Explicit Policy-conditioned Value Functionscited by 1
  30. 2025Apple: Toward General Active Perception via Reinforcement Learningcited by 1
  31. 2025On the Effect of Regularization in Policy Mirror Descentcited by 1
  32. 2025Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learningcited by 1
  33. 2025Understanding and Improving Hyperbolic Deep Reinforcement Learningcited by 1
  34. 2026RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learningcited by 1
  35. 2026RDA: Reward Design Agent for Reinforcement Learningcited by 1
  36. 2026Retry Policy Gradients in Continuous Action Spacescited by 1
  37. 2025Trajectory First: A Curriculum for Discovering Diverse Policiescited by 0
  38. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 0

50 most cited of 70 All 70 (JSON)