# Specializing Versatile Skill Libraries using Local Mixture of Experts

arXiv: https://arxiv.org/abs/2112.04216 · HTML: https://mlsanity.com/paper/2112.04216/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in MoE in reinforcement learning and robotics, one of 12 directions in Mixture-of-experts.
- Builds on 14 of its 38 references that are in the map.
- First posted on arXiv 8 Dec 2021.

## Placed in

- Mixture-of-experts / [MoE in reinforcement learning and robotics](https://mlsanity.com/moe/moe-in-reinforcement-learning-and-robotics/)

## Builds on (most cited first, 14 of 14)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills](https://mlsanity.com/paper/2002.03647/) (2020)
- [One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL](https://mlsanity.com/paper/2010.14484/) (2020)
- [Self-Paced Deep Reinforcement Learning](https://mlsanity.com/paper/2004.11812/) (2020)
- [Divide-and-Conquer Reinforcement Learning](https://mlsanity.com/paper/1711.09874/) (2017)
- [Learning Abstract Options](https://mlsanity.com/paper/1810.11583/) (2018)
- [Self-Paced Contextual Reinforcement Learning](https://mlsanity.com/paper/1910.02826/) (2019)
- [Discovering Diverse Solutions in Deep Reinforcement Learning by Maximizing State-Action-Based Mutual Information](https://mlsanity.com/paper/2103.07084/) (2021)
- [Trust-Region Variational Inference with Gaussian Mixture Models](https://mlsanity.com/paper/1907.04710/) (2019)
- [Expected Information Maximization: Using the I-Projection for Mixture Density Estimation](https://mlsanity.com/paper/2001.08682/) (2020)
- [Policy Search with High-Dimensional Context Variables](https://mlsanity.com/paper/1611.03231/) (2016)

## Built on by (most cited first, 24 of 24)

- [Theory on Mixture-of-Experts in Continual Learning](https://mlsanity.com/paper/2406.16437/) (2024)
- [Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations](https://mlsanity.com/paper/2402.14606/) (2024)
- [Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learning](https://mlsanity.com/paper/2412.12953/) (2024)
- [ProDMPs: A Unified Perspective on Dynamic and Probabilistic Movement Primitives](https://mlsanity.com/paper/2210.01531/) (2022)
- [Deep Black-Box Reinforcement Learning with Movement Primitives](https://mlsanity.com/paper/2210.09622/) (2022)
- [TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning](https://mlsanity.com/paper/2410.09536/) (2024)
- [Mixture of Experts (MoE): A Big Data Perspective](https://mlsanity.com/paper/2501.16352/) (2025)
- [Variational Distillation of Diffusion Policies into Mixture of Experts](https://mlsanity.com/paper/2406.12538/) (2024)
- [Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts](https://mlsanity.com/paper/2403.06966/) (2024)
- [Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning](https://mlsanity.com/paper/2401.11437/) (2024)
- [SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions](https://mlsanity.com/paper/2410.18416/) (2024)
- [Hierarchical Policy Blending as Inference for Reactive Robot Control](https://mlsanity.com/paper/2210.07890/) (2022)
- [MP3: Movement Primitive-Based (Re-)Planning Policy](https://mlsanity.com/paper/2306.12729/) (2023)
- [Constrained Ensemble Exploration for Unsupervised Skill Discovery](https://mlsanity.com/paper/2405.16030/) (2024)
- [Scalable and Efficient Continual Learning from Demonstration via a Hypernetwork-generated Stable Dynamics Model](https://mlsanity.com/paper/2311.03600/) (2023)
- [QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharing](https://mlsanity.com/paper/2302.00671/) (2023)
- [Skill Disentanglement for Imitation Learning from Suboptimal Demonstrations](https://mlsanity.com/paper/2306.07919/) (2023)
- [Hierarchical Policy Blending As Optimal Transport](https://mlsanity.com/paper/2212.01938/) (2022)
- [Bridging the gap between Learning-to-plan, Motion Primitives and Safe Reinforcement Learning](https://mlsanity.com/paper/2408.14063/) (2024)
- [Curriculum-Based Imitation of Versatile Skills](https://mlsanity.com/paper/2304.05171/) (2023)
- [Information Maximizing Curriculum: A Curriculum-Based Approach for Imitating Diverse Skills](https://mlsanity.com/paper/2303.15349/) (2023)
- [Revisiting Mixture Policies in Entropy-Regularized Actor-Critic](https://mlsanity.com/paper/2605.09157/) (2026)
- [Robot Skill Learning Via Classical Robotics-Based Generated Datasets: Advantages, Disadvantages, and Future Improvement](https://mlsanity.com/paper/2301.08794/) (2023)
- [MetaWorld-X: Hierarchical World Modeling via VLM-Orchestrated Experts for Humanoid Loco-Manipulation](https://mlsanity.com/paper/2603.08572/) (2026)

## Abstract (arXiv)

A long-cherished vision in robotics is to equip robots with skills that match the versatility and precision of humans. For example, when playing table tennis, a robot should be capable of returning the ball in various ways while precisely placing it at the desired location. A common approach to model such versatile behavior is to use a Mixture of Experts (MoE) model, where each expert is a contextual motion primitive. However, learning such MoEs is challenging as most objectives force the model to cover the entire context space, which prevents specialization of the primitives resulting in rather low-quality components. Starting from maximum entropy reinforcement learning (RL), we decompose the objective into optimizing an individual lower bound per mixture component. Further, we introduce a curriculum by allowing the components to focus on a local context region, enabling the model to learn highly accurate skill representations. To this end, we use local context distributions that are adapted jointly with the expert primitives. Our lower bound advocates an iterative addition of new components, where new components will concentrate on local context regions not covered by the current MoE. This local and incremental learning results in a modular MoE model of high accuracy and versatility, where both properties can be scaled by adding more components on the fly. We demonstrate this by an extensive ablation and on two challenging simulated robot skill learning tasks. We compare our achieved performance to LaDiPS and HiREPS, a known hierarchical policy search method for learning diverse skills.