mlsanityData to 7 Oct 2026

Mixtures of Experts Unlock Parameter Scaling for Deep RL

arXiv 2402.08609 ↗First posted 13 Feb 2024cs.LG cs.AIcited by 75

On the map: Mixture-of-experts → MoE in reinforcement learning and robotics

Abstract (arXiv)

The recent rapid progress in (self) supervised learning models is in large part predicted by empirical scaling laws: a model's performance scales proportionally to its size. Analogous scaling laws remain elusive for reinforcement learning domains, however, where increasing the parameter count of a model often hurts its final performance. In this paper, we demonstrate that incorporating Mixture-of-Expert (MoE) modules, and in particular Soft MoEs (Puigcerver et al., 2023), into value-based networks results in more parameter-scalable models, evidenced by substantial performance increases across a variety of training regimes and model sizes. This work thus provides strong empirical evidence towards developing scaling laws for reinforcement learning.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Placed in MoE in reinforcement learning and robotics, one of 12 directions in Mixture-of-experts. Builds on 54 of its 81 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

54 of 81 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  10. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  11. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  12. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
38 more
  1. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  2. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  3. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  4. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  5. 2019Model-Based Reinforcement Learning for Ataricited by 495
  6. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  7. 2019The State of Sparsity in Deep Neural Networkscited by 471
  8. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  9. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  10. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  11. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  12. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  13. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  14. 2020Revisiting Fundamentals of Experience Replaycited by 171
  15. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  16. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  17. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  18. 2020A Neural Dirichlet Process Mixture Model for Task-Free Continual Learningcited by 133
  19. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  20. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
  21. 2019When to use parametric models in reinforcement learning?cited by 118
  22. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 114
  23. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  24. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 79
  25. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  26. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  27. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  28. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  29. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  30. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  31. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
  32. 2020Scalable Transfer Learning with Expert Modelscited by 48
  33. 2023TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Expertscited by 45
  34. 2023Multi-Task Reinforcement Learning with Mixture of Orthogonal Expertscited by 45
  35. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 41
  36. 2023Small batch deep reinforcement learningcited by 35
  37. 2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
  38. 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 36; 2026: 18.

75 papers.

Sort:

By area:

  1. 2024One Step Diffusion via Shortcut Modelscited by 373
  2. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  3. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  4. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  5. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 70
  6. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
  7. 2024Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learningcited by 46
  8. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  9. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 39
  10. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 38
  11. 2024Generative AI Agents with Large Language Model for Satellite Networks via a Mixture of Experts Transmissioncited by 34
  12. 2024GeRM: A Generalist Robotic Model with Mixture-of-experts for Quadruped Robotcited by 28
38 more
  1. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  2. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 23
  3. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 22
  4. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
  5. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  6. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  7. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  8. 2024Neuroplastic Expansion in Deep Reinforcement Learningcited by 17
  9. 2025MoE-Loco: Mixture of Experts for Multitask Locomotioncited by 17
  10. 2025Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformercited by 15
  11. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14
  12. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
  13. 2025The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networkscited by 12
  14. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  15. 2024The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Designcited by 11
  16. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learningcited by 11
  17. 2026TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapsecited by 9
  18. 2024Active Fine-Tuning of Multi-Task Policiescited by 8
  19. 2025Yes, Q-learning Helps Offline In-Context RLcited by 8
  20. 2025Multi-Task Reinforcement Learning Enables Parameter Scalingcited by 8
  21. 2025Handling Delay in Real-Time Reinforcement Learningcited by 7
  22. 2025Mixture-of-Experts Meets In-Context Reinforcement Learningcited by 7
  23. 2025Impoola: The Power of Average Pooling for Image-Based Deep Reinforcement Learningcited by 6
  24. 2025Hadamax Encoding: Elevating Performance in Model-Free Ataricited by 6
  25. 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learningcited by 6
  26. 2024CALE: Continuous Arcade Learning Environmentcited by 5
  27. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  28. 2025Celo: Training Versatile Learned Optimizers on a Compute Dietcited by 4
  29. 2025Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learningcited by 4
  30. 2025Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learningcited by 4
  31. 2024SMOSE: Sparse Mixture of Shallow Experts for Interpretable Reinforcement Learning in Continuous Control Taskscited by 3
  32. 2025Convergence Rates for Softmax Gating Mixture of Expertscited by 3
  33. 2025One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planningcited by 3
  34. 2026Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environmentscited by 3
  35. 2025Combining Pre-Trained Models for Enhanced Feature Representation in Reinforcement Learningcited by 2
  36. 2025Sparsity-Driven Plasticity in Multi-Task Reinforcement Learningcited by 2
  37. 2026SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learningcited by 2
  38. 2024Hadamard Representation: Scaffolding Performance Across Model-free RLcited by 1

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
  2. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
  3. 2026Phase-Aware Mixture of Experts for Agentic Reinforcement Learning
  4. 2026How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization
  5. 2026Temporally Extended Mixture-of-Experts Models
  6. 2026Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
  7. 2026Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts
  8. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks