# Continuous Action Reinforcement Learning from a Mixture of Interpretable Experts

arXiv: https://arxiv.org/abs/2006.05911 · HTML: https://mlsanity.com/paper/2006.05911/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2020.

## Built on by (most cited first, 17 of 17)

- [Mixtures of Experts Unlock Parameter Scaling for Deep RL](https://mlsanity.com/paper/2402.08609/) (2024)
- [Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts](https://mlsanity.com/paper/2311.11385/) (2023)
- [Mixture of Experts in a Mixture of RL settings](https://mlsanity.com/paper/2406.18420/) (2024)
- [Mixture of Experts (MoE): A Big Data Perspective](https://mlsanity.com/paper/2501.16352/) (2025)
- [Variational Distillation of Diffusion Policies into Mixture of Experts](https://mlsanity.com/paper/2406.12538/) (2024)
- [Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer](https://mlsanity.com/paper/2505.24378/) (2025)
- [Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts](https://mlsanity.com/paper/2403.06966/) (2024)
- [Interaction Pattern Disentangling for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2207.03902/) (2022)
- [Hierarchical Policy Blending as Inference for Reactive Robot Control](https://mlsanity.com/paper/2210.07890/) (2022)
- [A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimization](https://mlsanity.com/paper/2403.11062/) (2024)
- [Hierarchical Policy Blending As Optimal Transport](https://mlsanity.com/paper/2212.01938/) (2022)
- [Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learning](https://mlsanity.com/paper/2408.00309/) (2024)
- [Prismatic World Model: Learning Compositional Dynamics for Planning in Hybrid Systems](https://mlsanity.com/paper/2512.08411/) (2025)
- [The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications](https://mlsanity.com/paper/2602.08019/) (2026)
- [Limits of Actor-Critic Algorithms for Decision Tree Policies Learning in IBMDPs](https://mlsanity.com/paper/2309.13365/) (2023)
- [Pessimistic Auxiliary Policy for Offline Reinforcement Learning](https://mlsanity.com/paper/2602.23974/) (2026)
- [Multi-Domain Learning with Global Expert Mapping](https://mlsanity.com/paper/2604.18842/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) has demonstrated its ability to solve high dimensional tasks by leveraging non-linear function approximators. However, these successes are mostly achieved by 'black-box' policies in simulated domains. When deploying RL to the real world, several concerns regarding the use of a 'black-box' policy might be raised. In order to make the learned policies more transparent, we propose in this paper a policy iteration scheme that retains a complex function approximator for its internal value predictions but constrains the policy to have a concise, hierarchical, and human-readable structure, based on a mixture of interpretable experts. Each expert selects a primitive action according to a distance to a prototypical state. A key design decision to keep such experts interpretable is to select the prototypical states from trajectory data. The main technical contribution of the paper is to address the challenges introduced by this non-differentiable prototypical state selection procedure. Experimentally, we show that our proposed algorithm can learn compelling policies on continuous action deep RL benchmarks, matching the performance of neural network based policies, but returning policies that are more amenable to human inspection than neural network or linear-in-feature policies.