mlsanityData to 14 Oct 2026

Mixture-of-Experts Meets In-Context Reinforcement Learning

arXiv 2506.05426 ↗First posted 5 Jun 2025cs.LG cs.AIcited by 8

On the map: Mixture-of-experts → MoE in reinforcement learning and robotics

also in 1 more
Abstract (arXiv)

In-context reinforcement learning (ICRL) has emerged as a promising paradigm for adapting RL agents to downstream tasks through prompt conditioning. However, two notable challenges remain in fully harnessing in-context learning within RL domains: the intrinsic multi-modality of the state-action-reward data and the diverse, heterogeneous nature of decision tasks. To tackle these challenges, we propose T2MIR (Token- and Task-wise MoE for In-context RL), an innovative framework that introduces architectural advances of mixture-of-experts (MoE) into transformer-based decision models. T2MIR substitutes the feedforward layer with two parallel layers: a token-wise MoE that captures distinct semantics of input tokens across multiple modalities, and a task-wise MoE that routes diverse tasks to specialized experts for managing a broad task distribution with alleviated gradient conflicts. To enhance task-wise routing, we introduce a contrastive learning method that maximizes the mutual information between the task and its router representation, enabling more precise capture of task-relevant information. The outputs of two MoE components are concatenated and fed into the next layer. Comprehensive experiments show that T2MIR significantly facilitates in-context learning capacity and outperforms various types of baselines. We bring the potential and promise of MoE to ICRL, offering a simple and scalable architectural enhancement to advance ICRL one step closer toward achievements in language and vision communities. Our code is available at https://github.com/NJU-RL/T2MIR.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in MoE in reinforcement learning and robotics, one of 12 directions in Mixture-of-experts. Builds on 63 of its 77 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

63 of 77 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer30 May
  2. 2024MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning19 Oct
  3. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RL13 Feb

Other references · 60

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
38 more
  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  2. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  5. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  6. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  8. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  9. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  10. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  11. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  12. 2025Learning to Reason under Off-Policy Guidancecited by 281
  13. 2023From Sparse to Soft Mixtures of Expertscited by 248
  14. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  15. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  16. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 146
  17. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 142
  18. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  19. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
  20. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 118
  21. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 111
  22. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 100
  23. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
  24. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 93
  25. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  26. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 55
  27. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 41
  28. 2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learningcited by 35
  29. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  30. 2023Emergence of In-Context Reinforcement Learning from Noise Distillationcited by 27
  31. 2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thoughtcited by 26
  32. 2023In-Context Reinforcement Learning for Variable Action Spacescited by 24
  33. 2020Lifelong Incremental Reinforcement Learning with Online Bayesian Inferencecited by 23
  34. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23
  35. 2023Context Shift Reduction for Offline Meta-Reinforcement Learningcited by 18
  36. 2024Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learningcited by 17
  37. 2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglementcited by 17
  38. 2024In-context Exploration-Exploitation for Reinforcement Learningcited by 16

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 7.

8 papers.

  1. 2025Diversity-Incentivized Exploration for Versatile Reasoningcited by 28
  2. 2026DyGRO-VLA: Cross-Task Scaling of Vision-Language-Action Models via Dynamic Grouped Residual Optimizationcited by 2
  3. 2026VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agentscited by 1
  4. 2026In-Context Reinforcement Learning under Non-Stationarity: A Surveycited by 1
  5. 2026Robust Exploration in Directed Controller Synthesis via Reinforcement Learning with Soft Mixture-of-Expertscited by 0
  6. 2026WiSDoM: Wireless Sparse Decision Transformer with Mixture-of-Experts for Multi-Task Mobile Network Optimizationcited by 0
  7. 2026MA-WAM: Multi-Agent World-Action Model for Test-Time Planningcited by 0
  8. 2026Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
  2. 2023Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts
  3. 2026Phase-Aware Mixture of Experts for Agentic Reinforcement Learning
  4. 2020Contextual Policy Transfer in Reinforcement Learning Domains via Deep Mixtures-of-Experts
  5. 2026RoutePack: Expert Placement and Attention-Aware Data Packing for MoE Reinforcement Learning
  6. 2026Expert-Space Exploration in MoE Reinforcement Learning
  7. 2026Vintix II: Decision Pre-Trained Transformer is a Scalable In-Context Reinforcement Learner
  8. 2026Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms