# Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts

arXiv: https://arxiv.org/abs/2311.11385 · HTML: https://mlsanity.com/paper/2311.11385/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 19 Nov 2023.

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks](https://mlsanity.com/paper/2306.13831/) (2023)
- [Distral: Robust Multitask Reinforcement Learning](https://mlsanity.com/paper/1707.04175/) (2017)
- [Multi-Task Reinforcement Learning with Context-based Representations](https://mlsanity.com/paper/2102.06177/) (2021)
- [Multi-Task Reinforcement Learning with Soft Modularization](https://mlsanity.com/paper/2003.13661/) (2020)
- [Multi-task Deep Reinforcement Learning with PopArt](https://mlsanity.com/paper/1809.04474/) (2018)
- [Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer](https://mlsanity.com/paper/1609.07088/) (2016)
- [Continual Learning in Low-rank Orthogonal Subspaces](https://mlsanity.com/paper/2010.11635/) (2020)
- [Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks](https://mlsanity.com/paper/1709.06079/) (2017)
- [Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform](https://mlsanity.com/paper/2002.01113/) (2020)
- [Orthogonal Deep Neural Networks](https://mlsanity.com/paper/1905.05929/) (2019)
- [MushroomRL: Simplifying Reinforcement Learning Research](https://mlsanity.com/paper/2001.01102/) (2020)
- [PaCo: Parameter-Compositional Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2210.11653/) (2022)
- [Orthogonal Machine Learning: Power and Limitations](https://mlsanity.com/paper/1711.00342/) (2017)
- [Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2104.09122/) (2021)
- [Continuous Action Reinforcement Learning from a Mixture of Interpretable Experts](https://mlsanity.com/paper/2006.05911/) (2020)
- [Optimization on Submanifolds of Convolution Kernels in CNNs](https://mlsanity.com/paper/1610.07008/) (2016)

## Built on by (most cited first, 30 of 49)

- [A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications](https://mlsanity.com/paper/2503.07137/) (2025)
- [Mixtures of Experts Unlock Parameter Scaling for Deep RL](https://mlsanity.com/paper/2402.08609/) (2024)
- [One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion](https://mlsanity.com/paper/2409.06366/) (2024)
- [Meta-World+: An Improved, Standardized, RL Benchmark](https://mlsanity.com/paper/2505.11289/) (2025)
- [Advancing Expert Specialization for Better MoE](https://mlsanity.com/paper/2505.22323/) (2025)
- [Load Balancing Mixture of Experts with Similarity Preserving Routers](https://mlsanity.com/paper/2506.14038/) (2025)
- [Mixture of Experts in a Mixture of RL settings](https://mlsanity.com/paper/2406.18420/) (2024)
- [Variational Distillation of Diffusion Policies into Mixture of Experts](https://mlsanity.com/paper/2406.12538/) (2024)
- [Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer](https://mlsanity.com/paper/2505.24378/) (2025)
- [Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts](https://mlsanity.com/paper/2403.06966/) (2024)
- [KungfuBot2: Learning Versatile Motion Skills for Humanoid Whole-Body Control](https://mlsanity.com/paper/2509.16638/) (2025)
- [Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks](https://mlsanity.com/paper/2507.23172/) (2025)
- [Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss](https://mlsanity.com/paper/2512.23447/) (2025)
- [Active Fine-Tuning of Multi-Task Policies](https://mlsanity.com/paper/2410.05026/) (2024)
- [Multi-Task Reinforcement Learning Enables Parameter Scaling](https://mlsanity.com/paper/2503.05126/) (2025)
- [One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2508.01561/) (2025)
- [Model-Based Transfer Learning for Contextual Reinforcement Learning](https://mlsanity.com/paper/2408.04498/) (2024)
- [Learning Policy Committees for Effective Personalization in MDPs with Diverse Tasks](https://mlsanity.com/paper/2503.01885/) (2025)
- [Plasticity-Aware Mixture of Experts for Learning Under QoE Shifts in Adaptive Video Streaming](https://mlsanity.com/paper/2504.09906/) (2025)
- [Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learning](https://mlsanity.com/paper/2506.04398/) (2025)
- [Interleaved Multitask Learning with Energy Modulated Learning Progress](https://mlsanity.com/paper/2504.00707/) (2025)
- [One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planning](https://mlsanity.com/paper/2509.07945/) (2025)
- [Continual Policy Consolidation for Lifelong Robot Learning](https://mlsanity.com/paper/2601.22475/) (2026)
- [Sparsity-Driven Plasticity in Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2508.06871/) (2025)
- [Prismatic World Model: Learning Compositional Dynamics for Planning in Hybrid Systems](https://mlsanity.com/paper/2512.08411/) (2025)
- [AnchorMoE: Interpretable Time Series Classification via Anchor-Routed MoE](https://mlsanity.com/paper/2606.03631/) (2026)
- [Nonparametric Bellman Mappings for Value Iteration in Distributed Reinforcement Learning](https://mlsanity.com/paper/2503.16192/) (2025)
- [Policy Search, Retrieval, and Composition via Task Similarity in Collaborative Agentic Systems](https://mlsanity.com/paper/2506.05577/) (2025)
- [Mixture-of-World Models: Scaling Multi-Task Reinforcement Learning with Modular Latent Dynamics](https://mlsanity.com/paper/2602.01270/) (2026)
- [STAIRS-Former: Spatio-Temporal Attention with Interleaved Recursive Structure Transformer for Offline Multi-task Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2603.11691/) (2026)

## Abstract (arXiv)

Multi-Task Reinforcement Learning (MTRL) tackles the long-standing problem of endowing agents with skills that generalize across a variety of problems. To this end, sharing representations plays a fundamental role in capturing both unique and common characteristics of the tasks. Tasks may exhibit similarities in terms of skills, objects, or physical properties while leveraging their representations eases the achievement of a universal policy. Nevertheless, the pursuit of learning a shared set of diverse representations is still an open challenge. In this paper, we introduce a novel approach for representation learning in MTRL that encapsulates common structures among the tasks using orthogonal representations to promote diversity. Our method, named Mixture Of Orthogonal Experts (MOORE), leverages a Gram-Schmidt process to shape a shared subspace of representations generated by a mixture of experts. When task-specific information is provided, MOORE generates relevant representations from this shared subspace. We assess the effectiveness of our approach on two MTRL benchmarks, namely MiniGrid and MetaWorld, showing that MOORE surpasses related baselines and establishes a new state-of-the-art result on MetaWorld.