mlsanityData to 14 Oct 2026

Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning

arXiv 2505.17749 ↗First posted 23 May 2025cs.LG cs.AIcited by 4

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Scaling deep reinforcement learning in pixel-based environments presents a significant challenge, often resulting in diminished performance. While recent works have proposed algorithmic and architectural approaches to address this, the underlying cause of the performance drop remains unclear. In this paper, we identify the connection between the output of the encoder (a stack of convolutional layers) and the ensuing dense layers as the main underlying factor limiting scaling capabilities; we denote this connection as the bottleneck, and we demonstrate that previous approaches implicitly target this bottleneck. As a result of our analyses, we present global average pooling as a simple yet effective way of targeting the bottleneck, thereby avoiding the complexity of earlier approaches.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 42 of its 51 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

42 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Hadamax Encoding: Elevating Performance in Model-Free Atari21 May
  2. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL2 Oct
  3. 2024Mixture of Experts in a Mixture of RL settings26 Jun
  4. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control25 May
  5. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RL6 Mar
  6. 2024In value-based deep reinforcement learning, a pruned network is a good network19 Feb
  7. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RL13 Feb

Other references · 35

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2016Layer Normalizationcited by 5,669
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  7. 2018DeepMind Control Suitecited by 956
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  10. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  11. 2013Network In Networkcited by 605
  12. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
23 more
  1. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  2. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  3. 2019Model-Based Reinforcement Learning for Ataricited by 502
  4. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  5. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  6. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  7. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  8. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  9. 2023From Sparse to Soft Mixtures of Expertscited by 248
  10. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  11. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  12. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  13. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  14. 2019When to use parametric models in reinforcement learning?cited by 118
  15. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 61
  16. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
  17. 2021Towards Robust Bisimulation Metric Learningcited by 49
  18. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  19. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 42
  20. 2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
  21. 2021Single-Shot Pruning for Offline Reinforcement Learningcited by 21
  22. 2023JaxPruner: A concise library for sparsity researchcited by 13
  23. 2025Impoola: The Power of Average Pooling for Image-Based Deep Reinforcement Learningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  2. 2026Gradient Iterated Temporal-Difference Learningcited by 0
  3. 2026Higher Resolution, Better Generalization: Unlocking Visual Scaling in Deep Reinforcement Learningcited by 0
  4. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning
  2. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies
  3. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning
  4. 2026Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning
  5. 2025Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning
  6. 2024Improving Generalization on the ProcGen Benchmark with Simple Architectural Changes and Scale
  7. 2025How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines
  8. 2025The Art of Scaling Reinforcement Learning Compute for LLMs