# Mix&Match - Agent Curricula for Reinforcement Learning

arXiv: https://arxiv.org/abs/1806.01780 · HTML: https://mlsanity.com/paper/1806.01780/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2018.

## Built on by (most cited first, 27 of 27)

- [Efficient Reinforcement Finetuning via Adaptive Curriculum Learning](https://mlsanity.com/paper/2504.05520/) (2025)
- [Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation](https://mlsanity.com/paper/2104.01655/) (2021)
- [Environment Generation for Zero-Shot Compositional Reinforcement Learning](https://mlsanity.com/paper/2201.08896/) (2022)
- [Improving Policy Optimization with Generalist-Specialist Learning](https://mlsanity.com/paper/2206.12984/) (2022)
- [Bayesian Generational Population-Based Training](https://mlsanity.com/paper/2207.09405/) (2022)
- [Asynchronous Multi-Agent Reinforcement Learning for Efficient Real-Time Multi-Robot Cooperative Exploration](https://mlsanity.com/paper/2301.03398/) (2023)
- [Knowledge Distillation Performs Partial Variance Reduction](https://mlsanity.com/paper/2305.17581/) (2023)
- [Self-Composing Policies for Scalable Continual Reinforcement Learning](https://mlsanity.com/paper/2506.14811/) (2025)
- [Cross-Episodic Curriculum for Transformer Agents](https://mlsanity.com/paper/2310.08549/) (2023)
- [Curriculum Design for Teaching via Demonstrations: Theory and Applications](https://mlsanity.com/paper/2106.04696/) (2021)
- [Behaviour-Diverse Automatic Penetration Testing: A Curiosity-Driven Multi-Objective Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2202.10630/) (2022)
- [Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution](https://mlsanity.com/paper/2404.04253/) (2024)
- [Toward Causal-Aware RL: State-Wise Action-Refined Temporal Difference](https://mlsanity.com/paper/2201.00354/) (2022)
- [The Curse of Diversity in Ensemble-Based Exploration](https://mlsanity.com/paper/2405.04342/) (2024)
- [Maximum Entropy Reinforcement Learning with Mixture Policies](https://mlsanity.com/paper/2103.10176/) (2021)
- [Investigating Value of Curriculum Reinforcement Learning in Autonomous Driving Under Diverse Road and Weather Conditions](https://mlsanity.com/paper/2103.07903/) (2021)
- [PoBRL: Optimizing Multi-Document Summarization by Blending Reinforcement Learning Policies](https://mlsanity.com/paper/2105.08244/) (2021)
- [Distilling Deep RL Models Into Interpretable Neuro-Fuzzy Systems](https://mlsanity.com/paper/2209.03357/) (2022)
- [FedTrans: Efficient Federated Learning via Multi-Model Transformation](https://mlsanity.com/paper/2404.13515/) (2024)
- [Distilling Privileged Information for Dubins Traveling Salesman Problems with Neighborhoods](https://mlsanity.com/paper/2404.16721/) (2024)
- [Continual Task Learning through Adaptive Policy Self-Composition](https://mlsanity.com/paper/2411.11364/) (2024)
- [Non-Markovian policies occupancy measures](https://mlsanity.com/paper/2205.13950/) (2022)
- [Augmenting Unsupervised Reinforcement Learning with Self-Reference](https://mlsanity.com/paper/2311.09692/) (2023)
- [Causality-Driven Reinforcement Learning for Joint Communication and Sensing](https://mlsanity.com/paper/2409.15329/) (2024)
- [Dynamic Action Interpolation: A Universal Approach for Accelerating Reinforcement Learning with Expert Guidance](https://mlsanity.com/paper/2504.18766/) (2025)
- [Inclusive Fitness as a Key Step Towards More Advanced Social Behaviors in Multi-Agent Reinforcement Learning Settings](https://mlsanity.com/paper/2510.12555/) (2025)
- [Selective Growing Q-Networks (SGQN): An Analysis of Adaptive Action Grids in Decoupled Q-Learning](https://mlsanity.com/paper/2610.20463/) (2026)

## Abstract (arXiv)

We introduce Mix&Match (M&M) - a training framework designed to facilitate rapid and effective learning in RL agents, especially those that would be too slow or too challenging to train otherwise. The key innovation is a procedure that allows us to automatically form a curriculum over agents. Through such a curriculum we can progressively train more complex agents by, effectively, bootstrapping from solutions found by simpler agents. In contradistinction to typical curriculum learning approaches, we do not gradually modify the tasks or environments presented, but instead use a process to gradually alter how the policy is represented internally. We show the broad applicability of our method by demonstrating significant performance gains in three different experimental setups: (1) We train an agent able to control more than 700 actions in a challenging 3D first-person task; using our method to progress through an action-space curriculum we achieve both faster training and better final performance than one obtains using traditional methods. (2) We further show that M&M can be used successfully to progress through a curriculum of architectural variants defining an agents internal state. (3) Finally, we illustrate how a variant of our method can be used to improve agent performance in a multitask setting.