mlsanityData to 13 Oct 2026

Hard Mixtures of Experts for Large Scale Weakly Supervised Vision

arXiv 1704.06363 ↗First posted 20 Apr 2017cs.CV stat.MLcited by 67

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Training convolutional networks (CNN's) that fit on a single GPU with minibatch stochastic gradient descent has become effective in practice. However, there is still no effective method for training large CNN's that do not fit in the memory of a few GPU cards, or for parallelizing CNN training. In this work we show that a simple hard mixture of experts model can be efficiently trained to good effect on large scale hashtag (multilabel) prediction tasks. Mixture of experts models are not new (Jacobs et. al. 1991, Collobert et. al. 2003), but in the past, researchers have had to devise sophisticated methods to deal with data fragmentation. We show empirically that modern weakly supervised data sets are large enough to support naive partitioning schemes where each data point is assigned to a single expert. Because the experts are independent, training them in parallel is easy, and evaluation is cheap for the size of the model. Furthermore, we show that we can use a single decoding layer for all the experts, allowing a unified feature embedding space. We demonstrate that it is feasible (and in fact relatively painless) to train far larger models than could be practically trained with standard CNN architectures, and that the extra capacity can be well used on current datasets.

Key facts. Cited by 67 papers in the map (data through 13 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 9 of its 21 references that are in the map. First posted on arXiv 20 Apr 2017.

Builds on

9 of 21 references.

References · 9

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,476
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,463
  4. 2016Identity Mappings in Deep Residual Networkscited by 1,962
  5. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  6. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  7. 2015Webly Supervised Learning of Convolutional Networkscited by 95
  8. 2016Network of Experts for Large-Scale Image Categorizationcited by 54
  9. 2014Self-informed neural network structure learningcited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 10; 2022: 14; 2023: 14; 2024: 14; 2025: 11; 2026: 3.

67 papers.

Sort:

By area:

  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 896
  2. 2021Scaling Vision with Sparse Mixture of Expertscited by 744
  3. 2022Mixture-of-Experts with Expert Choice Routingcited by 619
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 298
  5. 2021Hash Layers For Large Sparse Modelscited by 237
  6. 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
  7. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
  8. 2022Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolutioncited by 96
  9. 2021Generalizable Person Re-identification with Relevance-aware Mixture of Expertscited by 77
  10. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  11. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  12. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
38 more
  1. 2023Guiding Language Model Reasoning with Planning Tokenscited by 58
  2. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  3. 2022Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Expertscited by 50
  4. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Expertscited by 48
  5. 2022Residual Mixture of Expertscited by 39
  6. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Servicescited by 37
  7. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  8. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networkscited by 35
  9. 2023Brainformers: Trading Simplicity for Efficiencycited by 31
  10. 2023Robust Mixture-of-Expert Training for Convolutional Neural Networkscited by 30
  11. 2022Adaptive Mixture of Experts Learning for Generalizable Face Anti-Spoofingcited by 28
  12. 2023Enhancing NeRF akin to Enhancing LLMs: Generalizable NeRF Transformer with Mixture-of-View-Expertscited by 27
  13. 2021SpeechMoE2: Mixture-of-Experts Model with Improved Routingcited by 26
  14. 2024Routers in Vision Mixture of Experts: An Empirical Studycited by 22
  15. 2023Learning to Skip for Language Modelingcited by 19
  16. 2022NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Researchcited by 18
  17. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 17
  18. 2024Removal then Selection: A Coarse-to-Fine Fusion Perspective for RGB-Infrared Object Detectioncited by 16
  19. 2022Sparsely-gated Mixture-of-Expert Layers for CNN Interpretabilitycited by 14
  20. 2022Spatial Mixture-of-Expertscited by 14
  21. 2018Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithmscited by 13
  22. 2024Frequency-Adaptive Pan-Sharpening with Mixture of Expertscited by 13
  23. 2021Trust Your Robots! Predictive Uncertainty Estimation of Neural Networks with Sparse Gaussian Processescited by 12
  24. 2023Domain Generalization Using Large Pretrained Models with Mixture-of-Adapterscited by 11
  25. 2024Semi-Supervised Video Desnowing Network via Temporal Decoupling Experts and Distribution-Driven Contrastive Regularizationcited by 11
  26. 2023Revisiting Single-gated Mixtures of Expertscited by 10
  27. 2024DiPaCo: Distributed Path Compositioncited by 10
  28. 2025HAMoBE: Hierarchical and Adaptive Mixture of Biometric Experts for Video-based Person ReIDcited by 10
  29. 2024Multimodal Clinical Trial Outcome Prediction with Large Language Modelscited by 7
  30. 2024LLMBind: A Unified Modality-Task Integration Frameworkcited by 7
  31. 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Designcited by 6
  32. 20223M: Multi-loss, Multi-path and Multi-level Neural Networks for speech recognitioncited by 5
  33. 2024No Need to Talk: Asynchronous Mixture of Language Modelscited by 5
  34. 2025A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusioncited by 5
  35. 2021Large-Scale Attribute-Object Compositionscited by 4
  36. 2024Need a Small Specialized Language Model? Plan Early!cited by 4
  37. 2021Mixtures of Deep Neural Experts for Automated Speech Scoringcited by 3
  38. 2022Towards Efficient Single Image Dehazing and Desnowingcited by 3

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
  2. 2024Multi-Task Dense Prediction via Mixture of Low-Rank Experts
  3. 2024Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Experts
  4. 2023Scaling Vision-Language Models with Sparse Mixture of Experts
  5. 2025Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks
  6. 2026Mixture of Experts with Soft Nearest Neighbor Loss: Resolving Expert Collapse via Representation Disentanglement
  7. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  8. 2026Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization