mlsanityData to 7 Oct 2026

Mixture of Experts in a Mixture of RL settings

arXiv 2406.18420 ↗First posted 26 Jun 2024cs.LG cs.AIcited by 21

On the map: Mixture-of-experts → MoE in reinforcement learning and robotics

Abstract (arXiv)

Mixtures of Experts (MoEs) have gained prominence in (self-)supervised learning due to their enhanced inference efficiency, adaptability to distributed training, and modularity. Previous research has illustrated that MoEs can significantly boost Deep Reinforcement Learning (DRL) performance by expanding the network's parameter count while reducing dormant neurons, thereby enhancing the model's learning capacity and ability to deal with non-stationarity. In this work, we shed more light on MoEs' ability to deal with non-stationarity and investigate MoEs in DRL settings with "amplified" non-stationarity via multi-task training, providing further evidence that MoEs improve learning capacity. In contrast to previous work, our multi-task results allow us to better understand the underlying causes for the beneficial effect of MoE in DRL training, the impact of the various MoE components, and insights into how best to incorporate them in actor-critic-based DRL networks. Finally, we also confirm results from previous work.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in MoE in reinforcement learning and robotics, one of 12 directions in Mixture-of-experts. Builds on 59 of its 84 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

59 of 84 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RL13 Feb
  2. 2021Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learning19 Apr

Other references · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  11. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  12. 2019The State of Sparsity in Deep Neural Networkscited by 471
38 more
  1. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  2. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  3. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  4. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  5. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  6. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  7. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  8. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  9. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  10. 2023Understanding plasticity in neural networkscited by 170
  11. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  12. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  13. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  14. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  15. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 118
  16. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 118
  17. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  18. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 114
  19. 2018Model-Based Active Explorationcited by 105
  20. 2022Discovered Policy Optimisationcited by 103
  21. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 79
  22. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  23. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  24. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  25. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  26. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  27. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
  28. 2023TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Expertscited by 45
  29. 2023Multi-Task Reinforcement Learning with Mixture of Orthogonal Expertscited by 45
  30. 2022Model-Free Opponent Shapingcited by 43
  31. 2022COLA: Consistent Learning with Opponent-Learning Awarenesscited by 38
  32. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 38
  33. 2023Small batch deep reinforcement learningcited by 35
  34. 2022Pruning's Effect on Generalization Through the Lens of Training and Regularizationcited by 31
  35. 2021Ensemble Bootstrapping for Q-Learningcited by 30
  36. 2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
  37. 2021Single-Shot Pruning for Offline Reinforcement Learningcited by 21
  38. 2020Biased Mixtures Of Experts: Enabling Computer Vision Inference Under Data Transfer Limitationscited by 20

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 10.

21 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  2. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 22
  3. 2024Neuroplastic Expansion in Deep Reinforcement Learningcited by 17
  4. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14
  5. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
  6. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  7. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  8. 2025Plasticity-Aware Mixture of Experts for Learning Under QoE Shifts in Adaptive Video Streamingcited by 4
  9. 2025Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learningcited by 4
  10. 2026SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learningcited by 2
  11. 2026Plasticity-Enhanced Multi-Agent Mixture of Experts for Dynamic Objective Adaptation in UAVs-Assisted Emergency Communication Networkscited by 1
  12. 2026Temporal Recurrence Favors Fewer Layerscited by 1
9 more
  1. 2025Topology-Assisted Spatio-Temporal Pattern Disentangling for Scalable MARL in Large-scale Autonomous Traffic Controlcited by 0
  2. 2025Dynamic Mixture of Experts Against Severe Distribution Shiftscited by 0
  3. 2025RAST-MoE-RL: A Regime-Aware Spatio-Temporal MoE Framework for Deep Reinforcement Learning in Ride-Hailingcited by 0
  4. 2026Robust Exploration in Directed Controller Synthesis via Reinforcement Learning with Soft Mixture-of-Expertscited by 0
  5. 2026Higher Resolution, Better Generalization: Unlocking Visual Scaling in Deep Reinforcement Learningcited by 0
  6. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitionscited by 0
  7. 2026Multi-scale Mixture of World Models for Embodied Agents in Evolving Environmentscited by 0
  8. 2026PRIME: Plasticity Recovery in Multi-Agent Environments for UAV-Assisted Emergency Communication Networkscited by 0
  9. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
  2. 2022Towards Understanding Mixture of Experts in Deep Learning
  3. 2026Phase-Aware Mixture of Experts for Agentic Reinforcement Learning
  4. 2023Improving Expert Specialization in Mixture of Experts
  5. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms
  6. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks
  7. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  8. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Experts