mlsanityData to 7 Oct 2026

Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learning

arXiv 2104.09122 ↗First posted 19 Apr 2021cs.LG cs.AIcited by 26

On the map: Mixture-of-experts → MoE in reinforcement learning and robotics

Abstract (arXiv)

Deep reinforcement learning (DRL) has successfully solved various problems recently, typically with a unimodal policy representation. However, grasping distinguishable skills for some tasks with non-unique optima can be essential for further improving its learning efficiency and performance, which may lead to a multimodal policy represented as a mixture-of-experts (MOE). To our best knowledge, present DRL algorithms for general utility do not deploy this method as policy function approximators due to the potential challenge in its differentiability for policy learning. In this work, we propose a probabilistic mixture-of-experts (PMOE) implemented with a Gaussian mixture model (GMM) for multimodal policy, together with a novel gradient estimator for the indifferentiability problem, which can be applied in generic off-policy and on-policy DRL algorithms using stochastic policies, e.g., Soft Actor-Critic (SAC) and Proximal Policy Optimisation (PPO). Experimental results testify the advantage of our method over unimodal polices and two different MOE methods, as well as a method of option frameworks, based on the above two types of DRL algorithms, on six MuJoCo tasks. Different gradient estimations for GMM like the reparameterisation trick (Gumbel-Softmax) and the score-ratio trick are also compared with our method. We further empirically demonstrate the distinguishable primitives learned with PMOE and show the benefits of our method in terms of exploration.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in MoE in reinforcement learning and robotics, one of 12 directions in Mixture-of-experts. Builds on 26 of its 56 references that are in the map. First posted on arXiv 19 Apr 2021.

Builds on

26 of 56 references.

References · 26

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  9. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  10. 2016OpenAI Gymcited by 1,684
  11. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  12. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
14 more
  1. 2016The Option-Critic Architecturecited by 433
  2. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  3. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  4. 2014Neural Variational Inference and Learning in Belief Networkscited by 116
  5. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  6. 2016Variational inference for Monte Carlo objectivescited by 83
  7. 2017REBAR: Low-variance, unbiased gradient estimates for discrete latent variable modelscited by 78
  8. 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 77
  9. 2015Deep Reinforcement Learning in Parameterized Action Spacecited by 65
  10. 2013Deep AutoRegressive Networkscited by 60
  11. 2015MuProp: Unbiased Backpropagation for Stochastic Neural Networkscited by 36
  12. 2016Combining policy gradient and Q-learningcited by 23
  13. 2020Approximation Based Variance Reduction for Reparameterization Gradientscited by 10
  14. 2017The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 7; 2024: 5; 2025: 5; 2026: 5.

26 papers.

Sort:

By area:

  1. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93
  2. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  3. 2023Towards Generalizable Neural Solvers for Vehicle Routing Problems via Ensemble with Transferrable Local Policycited by 61
  4. 2023Multi-Task Reinforcement Learning with Mixture of Orthogonal Expertscited by 45
  5. 2024Generative AI Agents with Large Language Model for Satellite Networks via a Mixture of Experts Transmissioncited by 34
  6. 2023PECAN: Leveraging Policy Ensemble for Context-Aware Zero-Shot Human-AI Coordinationcited by 25
  7. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Expertscited by 23
  8. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  9. 2022Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemblecited by 18
  10. 2024Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Expertscited by 14
  11. 2024Variational Distillation of Diffusion Policies into Mixture of Expertscited by 14
  12. 2022Latent State Marginalization as a Low-cost Approach for Improving Explorationcited by 13
14 more
  1. 2023Wasserstein Gradient Flows for Optimizing Gaussian Mixture Policiescited by 8
  2. 2022ASPiRe:Adaptive Skill Priors for Reinforcement Learningcited by 7
  3. 2023Improving Deep Policy Gradients with Value Function Searchcited by 7
  4. 2024SMOSE: Sparse Mixture of Shallow Experts for Interpretable Reinforcement Learning in Continuous Control Taskscited by 3
  5. 2025Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularizationcited by 3
  6. 2025Prismatic World Model: Learning Compositional Dynamics for Planning in Hybrid Systemscited by 2
  7. 2022Pluralistic Image Completion with Probabilistic Mixture-of-Expertscited by 1
  8. 2025Wasserstein Barycenter Soft Actor-Criticcited by 1
  9. 2026MePoly: Max Entropy Polynomial Policy Optimizationcited by 1
  10. 2026Revisiting Mixture Policies in Entropy-Regularized Actor-Criticcited by 1
  11. 2025RAST-MoE-RL: A Regime-Aware Spatio-Temporal MoE Framework for Deep Reinforcement Learning in Ride-Hailingcited by 0
  12. 2026Moment Matching Q-Learningcited by 0
  13. 2026Reinforcement Learning: From Algorithms To Foundation Modelscited by 0
  14. 2026Clustered Randomized Smoothing for Stochastic Prediction Functionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Contextual Policy Transfer in Reinforcement Learning Domains via Deep Mixtures-of-Experts
  2. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RL
  3. 2020Off-policy Maximum Entropy Reinforcement Learning : Soft Actor-Critic with Advantage Weighted Mixture Policy(SAC-AWMP)
  4. 2025Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
  5. 2026Phase-Aware Mixture of Experts for Agentic Reinforcement Learning
  6. 2026SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learning
  7. 2026Variational Proximal Policy Optimization
  8. 2019Mo"ET: Mixture of Expert Trees and its Application to Verifiable Reinforcement Learning