mlsanityData to 14 Oct 2026

1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities

arXiv 2503.14858 ↗First posted 19 Mar 2025cs.LG cs.AIcited by 45

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Scaling up self-supervised learning has driven breakthroughs in language and vision, yet comparable progress has remained elusive in reinforcement learning (RL). In this paper, we study building blocks for self-supervised RL that unlock substantial improvements in scalability, with network depth serving as a critical factor. Whereas most RL papers in recent years have relied on shallow architectures (around 2 - 5 layers), we demonstrate that increasing the depth up to 1024 layers can significantly boost performance. Our experiments are conducted in an unsupervised goal-conditioned setting, where no demonstrations or rewards are provided, so an agent must explore (from scratch) and learn how to maximize the likelihood of reaching commanded goals. Evaluated on simulated locomotion and manipulation tasks, our approach increases performance on the self-supervised contrastive RL algorithm by 2× - 50×, outperforming other goal-conditioned baselines. Increasing the model depth not only increases success rates but also qualitatively changes the behaviors learned. The project webpage and code can be found here: https://wang-kevin3290.github.io/scaling-crl/.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 52 of its 67 references that are in the map. First posted on arXiv 19 Mar 2025.

Builds on

52 of 67 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning13 Oct
  2. 2024Accelerating Goal-Conditioned RL Algorithms and Research20 Aug
  3. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making24 Jun
  4. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control25 May
  5. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RL6 Mar
  6. 2024Disentangling the Causes of Plasticity Loss in Neural Networks29 Feb
  7. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RL13 Feb

Other references · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2016Layer Normalizationcited by 5,669
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  9. 2022Emergent Abilities of Large Language Modelscited by 2,512
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
33 more
  1. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  2. 2017Hindsight Experience Replaycited by 1,136
  3. 2021Scaling Vision Transformerscited by 999
  4. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  6. 2017Deep Reinforcement Learning that Matterscited by 754
  7. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  8. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  9. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  10. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  11. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  12. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  13. 2022Multi-Game Decision Transformerscited by 214
  14. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  15. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  16. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  17. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  18. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  19. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
  20. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 103
  21. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 73
  22. 2024How Does Critical Batch Size Scale in Pre-training?cited by 73
  23. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 70
  24. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 63
  25. 2017Multi-level Residual Networks from Dynamical Systems Viewcited by 59
  26. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  27. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  28. 2021Training Larger Networks for Deep Reinforcement Learningcited by 38
  29. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Modelcited by 38
  30. 2023Small batch deep reinforcement learningcited by 36
  31. 2023Contrastive Difference Predictive Codingcited by 36
  32. 2022Metric Residual Networks for Sample Efficient Goal-Conditioned Reinforcement Learningcited by 13
  33. 2023Scaling Laws for Imitation Learning in Single-Agent Gamescited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 36.

45 papers.

Sort:

By area:

  1. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
  2. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 22
  3. 2026mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterationscited by 17
  4. 2025The Serial Scaling Hypothesiscited by 13
  5. 2025Flattening Hierarchies with Policy Bootstrappingcited by 11
  6. 2026TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapsecited by 9
  7. 2026GIANTS: Generative Insight Anticipation from Scientific Literaturecited by 8
  8. 2025Compute-Optimal Scaling for Value-Based Deep RLcited by 6
  9. 2025Do Depth-Grown Models Overcome the Curse of Depth? An In-Depth Analysiscited by 6
  10. 2026ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scalingcited by 5
  11. 2025Is Temporal Difference Learning the Gold Standard for Stitching in RL?cited by 4
  12. 2026On the Role of Computation in Reinforcement Learningcited by 3
33 more
  1. 2026Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environmentscited by 3
  2. 2026SVL: Goal-Conditioned Reinforcement Learning as Survival Learningcited by 3
  3. 2026Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RLcited by 3
  4. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learningcited by 2
  5. 2026Survival Reinforcement Learning: Toward Scalable Self-Supervised RLcited by 2
  6. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 1
  7. 2025Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learningcited by 1
  8. 2026Quantile-Coupled Flow Matching for Distributional Reinforcement Learningcited by 1
  9. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  10. 2026Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  11. 2026Inverting the Bellman Equation: From Q-Values to World Modelscited by 1
  12. 2026Temporal Recurrence Favors Fewer Layerscited by 1
  13. 2026Model-Free DRL Control for Power Inverters: From Policy Learning to Real-Time Implementation via Knowledge Distillationcited by 0
  14. 2026Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learningcited by 0
  15. 2026Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisationcited by 0
  16. 2026Adaptive Outer-Loop Control of Quadrotors via Reinforcement Learningcited by 0
  17. 2026MATE: Solving Contextual Markov Decision Processes with Memory of Accumulated Transition Embeddingscited by 0
  18. 2026Scalable Reinforcement Learning via Adaptive Batch Scalingcited by 0
  19. 2026SWIM: Compact Environment Representation for Reinforcement Learning Human Swimmingcited by 0
  20. 2026Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learningcited by 0
  21. 2026Bridging Reinforcement Learning and Optimal Control via Feasible Action Mappingcited by 0
  22. 2026Arrive and Survive: Scaling Safe Goal-Conditioned Policy Learning from One-Bit Failure Signalscited by 0
  23. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learningcited by 0
  24. 2026Mind the Phase: Effective Rank and Representation Health in Legged Locomotioncited by 0
  25. 2026OneBid: A Unified Auto-Bidding Foundation Model for Diverse oCPX Advertising Scenarioscited by 0
  26. 2026FlexLoop: Depth-Elastic Looped Policies for Adaptive Test-Time Computation in Deep RLcited by 0
  27. 2026ChronoSRL: Temporal Geometry for Self-Supervised Reinforcement Learningcited by 0
  28. 2026Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPOcited by 0
  29. 2026In-Distribution Imagination for Model-Based Offline Reinforcement Learningcited by 0
  30. 2026Direction-Conditioned Policies for Online Goal-Conditioned Reinforcement Learningcited by 0
  31. 2026Imagining Goals with Geometric Horizon Models for Policy Generalizationcited by 0
  32. 2026Outperforming 1000 Euclidean Layers With Hyperbolic Goal-Conditioned RLcited by 0
  33. 2026Hyperspherical Contrastive Representations Unlock Planning and Explorationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
  2. 2025Discover, Learn, and Reinforce: Scaling Vision-Language-Action Pretraining with Diverse RL-Generated Trajectories
  3. 2025SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models
  4. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes
  5. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning
  6. 2026MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement
  7. 2025The Art of Scaling Reinforcement Learning Compute for LLMs
  8. 2025Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Following