mlsanityData to 7 Oct 2026

Residual Mixture of Experts

arXiv 2204.09636 ↗First posted 20 Apr 2022cs.CVcited by 39

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Mixture of Experts (MoE) is able to scale up vision transformers effectively. However, it requires prohibiting computation resources to train a large MoE transformer. In this paper, we propose Residual Mixture of Experts (RMoE), an efficient training pipeline for MoE vision transformers on downstream tasks, such as segmentation and detection. RMoE achieves comparable results with the upper-bound MoE training, while only introducing minor additional training cost than the lower-bound non-MoE training pipelines. The efficiency is supported by our key observation: the weights of an MoE transformer can be factored into an input-independent core and an input-dependent residual. Compared with the weight core, the weight residual can be efficiently trained with much less computation resource, e.g., finetuning on the downstream data. We show that, compared with the current MoE training pipeline, we get comparable results while saving over 30% training cost. When compared with state-of-the-art non- MoE transformers, such as Swin-T / CvT-13 / Swin-L, we get +1.1 / 0.9 / 1.0 mIoU gain on ADE20K segmentation and +1.4 / 1.6 / 0.6 AP gain on MS-COCO object detection task with less than 3% additional training cost.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 45 of its 65 references that are in the map. First posted on arXiv 20 Apr 2022.

Builds on

45 of 65 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Scaling Vision with Sparse Mixture of Experts10 Jun
  2. 2020Biased Mixtures Of Experts: Enabling Computer Vision Inference Under Data Transfer Limitations21 Aug
  3. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Vision20 Apr

Other references · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2016Densely Connected Convolutional Networkscited by 6,691
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  11. 2017Squeeze-and-Excitation Networkscited by 4,763
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
30 more
  1. 2014Going Deeper with Convolutionscited by 4,231
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  9. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  10. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  12. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  13. 2021Scaling Vision Transformerscited by 993
  14. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  15. 2021Going deeper with Image Transformerscited by 759
  16. 2021Transformer in Transformercited by 632
  17. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  18. 2019Hybrid Task Cascade for Instance Segmentationcited by 479
  19. 2021TransReID: Transformer-based Object Re-Identificationcited by 432
  20. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  21. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  22. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  23. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  24. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 110
  25. 2021Training Vision Transformers for Image Retrievalcited by 90
  26. 2015Distributed Gaussian Processescited by 84
  27. 2016Stochastic Multiple Choice Learning for Training Diverse Deep Ensemblescited by 77
  28. 2021Firefly Neural Architecture Descent: a General Approach for Growing Neural Networkscited by 48
  29. 2015Generative Image Modeling Using Spatial LSTMscited by 35
  30. 2017Confident Multiple Choice Learningcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 9; 2024: 11; 2025: 8; 2026: 5.

39 papers.

Sort:

By area:

  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  2. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  3. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  4. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  5. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  6. 2023Brainformers: Trading Simplicity for Efficiencycited by 31
  7. 2024Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Expertscited by 31
  8. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorizationcited by 28
  9. 2025SkySense V2: A Unified Foundation Model for Multi-modal Remote Sensingcited by 26
  10. 2023Octavius: Mitigating Task Interference in MLLMs via LoRA-MoEcited by 25
  11. 2024Routers in Vision Mixture of Experts: An Empirical Studycited by 22
  12. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
27 more
  1. 2025LiMoE: Mixture of LiDAR Representation Learners from Automotive Scenescited by 19
  2. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  3. 2024Task-customized Masked AutoEncoder via Mixture of Cluster-conditional Expertscited by 16
  4. 2022Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learnerscited by 15
  5. 2023HQ-50K: A Large-scale, High-quality Dataset for Image Restorationcited by 15
  6. 2022MoEC: Mixture of Expert Clusterscited by 14
  7. 2022Sparsely-gated Mixture-of-Expert Layers for CNN Interpretabilitycited by 13
  8. 2023Alternating Updates for Efficient Transformerscited by 13
  9. 2025Steepest Descent Density Control for Compact 3D Gaussian Splattingcited by 13
  10. 2024UniRestorer: Universal Image Restoration via Adaptively Estimating Image Degradation at Proper Granularitycited by 12
  11. 2023WM-MoE: Weather-aware Multi-scale Mixture-of-Experts for Blind Adverse Weather Removalcited by 11
  12. 2024textttMoE-RBench: Towards Building Reliable Language Models with Sparse Mixture-of-Expertscited by 7
  13. 2025YOLO-Master: MOE-Accelerated with Specialized Transformers for Enhanced Real-time Detectioncited by 4
  14. 2022RepMode: Learning to Re-parameterize Diverse Experts for Subcellular Structure Predictioncited by 3
  15. 2023COMET: Learning Cardinality Constrained Mixture of Experts with Trees and Local Searchcited by 2
  16. 2025One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learningcited by 2
  17. 2025TAP: Two-Stage Adaptive Personalization of Multi-Task and Multi-Modal Foundation Models in Federated Learningcited by 2
  18. 2023Reinforcement Learning-based Mixture of Vision Transformers for Video Violence Recognitioncited by 1
  19. 2024ViMoE: An Empirical Study of Designing Vision Mixture-of-Expertscited by 1
  20. 2024Mr. DETR++: Instructive Multi-Route Training for Detection Transformers with Mixture-of-Expertscited by 1
  21. 2026MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understandingcited by 1
  22. 2023The Power of External Memory in Increasing Predictive Model Capacitycited by 0
  23. 2025MultiPL-MoE: Multi-Programming-Lingual Extension of Large Language Models through Hybrid Mixture-of-Expertscited by 0
  24. 2026ExFusion: Efficient Transformer Training via Multi-Experts Fusioncited by 0
  25. 2026Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Expertscited by 0
  26. 2026SP-MoMamba: Superpixel-driven Mixture of State Space Experts for Efficient Image Super-Resolutioncited by 0
  27. 2026Unifying Semantic Priors and High-Frequency Traces: Enhancing V-JEPA with Mixture-of-Experts for Robust Synthetic Image Forensicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024How Lightweight Can A Vision Transformer Be
  2. 2023Experts Weights Averaging: A New General Training Scheme for Vision Transformers
  3. 2025Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks
  4. 2024Mixture of Experts in Image Classification: What's the Sweet Spot?
  5. 2026CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selection
  6. 2023From Sparse to Soft Mixtures of Experts
  7. 2023Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Experts
  8. 2025Dense Backpropagation Improves Training for Sparse Mixture-of-Experts