mlsanityData to 7 Oct 2026

Scaling Vision with Sparse Mixture of Experts

arXiv 2106.05974 ↗First posted 10 Jun 2021cs.CV cs.LG stat.MLcited by 740

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Sparsely-gated Mixture of Experts networks (MoEs) have demonstrated excellent scalability in Natural Language Processing. In Computer Vision, however, almost all performant networks are "dense", that is, every input is processed by every parameter. We present a Vision MoE (V-MoE), a sparse version of the Vision Transformer, that is scalable and competitive with the largest dense networks. When applied to image recognition, V-MoE matches the performance of state-of-the-art networks, while requiring as little as half of the compute at inference time. Further, we propose an extension to the routing algorithm that can prioritize subsets of each input across the entire batch, leading to adaptive per-image compute. This allows V-MoE to trade-off performance and compute smoothly at test-time. Finally, we demonstrate the potential of V-MoE to scale vision models, and train a 15B parameter model that attains 90.35% on ImageNet.

Key facts. Cited by 740 papers in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 43 of its 75 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

43 of 75 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Biased Mixtures Of Experts: Enabling Computer Vision Inference Under Data Transfer Limitations21 Aug
  2. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Vision20 Apr

Other references · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2018Deep contextualized word representationscited by 2,285
  11. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  12. 2013Describing Textures in the Wildcited by 2,110
29 more
  1. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  2. 2021ViViT: A Video Vision Transformercited by 1,610
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  4. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  7. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  8. 2021Scaling Vision Transformerscited by 993
  9. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  10. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  11. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  12. 2021Going deeper with Image Transformerscited by 759
  13. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  14. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  15. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  16. 2020Meta Pseudo Labelscited by 372
  17. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  18. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  19. 2019A Baseline for Few-Shot Image Classificationcited by 298
  20. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  21. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  22. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  23. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 125
  24. 2013Deep Learning of Representations: Looking Forwardcited by 84
  25. 2016Network of Experts for Large-Scale Image Categorizationcited by 54
  26. 2019Routing Networks and the Challenges of Modular and Compositional Computationcited by 49
  27. 2013Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networkscited by 44
  28. 2014Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learningcited by 24
  29. 2014Deep Sequential Neural Networkcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 71; 2023: 108; 2024: 176; 2025: 191; 2026: 178.

740 papers.

Sort:

By area:

  1. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  2. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  3. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  5. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  6. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  7. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  8. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  9. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  10. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 376
  11. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  12. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 280
38 more
  1. 2023From Sparse to Soft Mixtures of Expertscited by 245
  2. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  3. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  4. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  5. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  6. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  7. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  8. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  9. 2022FourCastNet: Accelerating Global High-Resolution Weather Forecasting using Adaptive Fourier Neural Operatorscited by 188
  10. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  11. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  12. 2022DaViT: Dual Attention Vision Transformerscited by 174
  13. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  14. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  15. 2022Advancing Plain Vision Transformer Towards Remote Sensing Foundation Modelcited by 149
  16. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  17. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  18. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 141
  19. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  20. 2021A Survey of Visual Transformerscited by 117
  21. 2021MVSS-Net: Multi-View Multi-Scale Supervised Networks for Image Manipulation Detectioncited by 117
  22. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114
  23. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  24. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  25. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 110
  26. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  27. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  28. 2022HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutionscited by 103
  29. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  30. 2025UMA: A Family of Universal Models for Atomscited by 99
  31. 2021The Efficiency Misnomercited by 98
  32. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  33. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  34. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 87
  35. 2025ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulationcited by 87
  36. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  37. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  38. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84

50 most cited of 740 All 740 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization
  2. 2024How Lightweight Can A Vision Transformer Be
  3. 2023Mixture of Tokens: Continuous MoE through Cross-Example Aggregation
  4. 2025Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
  5. 2025LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference
  6. 2026AxMoE: Characterizing the Impact of Approximate Multipliers on Mixture-of-Experts DNN Architectures
  7. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
  8. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks