mlsanityData to 7 Oct 2026

Sparse MoEs meet Efficient Ensembles

arXiv 2110.03360 ↗First posted 7 Oct 2021cs.LG cs.CV stat.MLcited by 19

Not on the map. Why?

Abstract (arXiv)

Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, often exhibit strong performance compared to individual models. We study the interplay of two popular classes of such models: ensembles of neural networks and sparse mixture of experts (sparse MoEs). First, we show that the two approaches have complementary features whose combination is beneficial. This includes a comprehensive evaluation of sparse MoEs in uncertainty related benchmarks. Then, we present Efficient Ensemble of Experts (E3), a scalable and simple ensemble of sparse MoEs that takes the best of both classes of models, while using up to 45% fewer FLOPs than a deep ensemble. Extensive experiments demonstrate the accuracy, log-likelihood, few-shot learning, robustness, and uncertainty improvements of E3 over several challenging vision Transformer-based baselines. E3 not only preserves its efficiency while scaling to models with up to 2.7B parameters, but also provides better predictive performance and uncertainty estimates for larger models.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 54 of its 102 references that are in the map. First posted on arXiv 7 Oct 2021.

Builds on

54 of 102 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  8. 2017On Calibration of Modern Neural Networkscited by 4,860
  9. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  10. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
38 more
  1. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  2. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  3. 2013Describing Textures in the Wildcited by 2,110
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2019Natural Adversarial Examplescited by 1,401
  8. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  9. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  10. 2021Scaling Vision Transformerscited by 993
  11. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  12. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  13. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  14. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  15. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  16. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  17. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  18. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  19. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  20. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  21. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  22. 2021Vision Transformers are Robust Learnerscited by 208
  23. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  24. 2019Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Visioncited by 182
  25. 2020Training independent subnetworks for robust predictioncited by 177
  26. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  27. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 149
  28. 2015Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networkscited by 138
  29. 2021Towards Robust Vision Transformercited by 137
  30. 2021Bayesian Neural Network Priors Revisitedcited by 121
  31. 2021Repulsive Deep Ensembles are Bayesiancited by 103
  32. 2020Bayesian Deep Learning via Subnetwork Inferencecited by 85
  33. 2020Learning with Differentiable Perturbed Optimizerscited by 81
  34. 2020Depth Uncertainty in Neural Networkscited by 56
  35. 2021Go Wider Instead of Deepercited by 56
  36. 2019Analyzing the Role of Model Uncertainty for Electronic Health Recordscited by 41
  37. 2021MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworkscited by 32
  38. 2020Hydra: Preserving Ensemble Diversity for Model Distillationcited by 31

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 5; 2024: 7; 2025: 0; 2026: 1.

19 papers.

Sort:

By area:

  1. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  2. 2023A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Modelscited by 39
  3. 2022On the Adversarial Robustness of Mixture of Expertscited by 27
  4. 2024Mixture of Nested Experts: Adaptive Processing of Visual Tokenscited by 25
  5. 2021Transferability Metrics for Selecting Source Model Ensemblescited by 24
  6. 2024Routers in Vision Mixture of Experts: An Empirical Studycited by 22
  7. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Expertscited by 21
  8. 2024FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusioncited by 10
  9. 2024Exploring Forgetting in Large Language Model Pre-Trainingcited by 9
  10. 2022A Theoretical View on Sparsely Activated Networkscited by 6
  11. 2023Associative Transformercited by 4
  12. 2023Embedding Space Interpolation Beyond Mini-Batch, Beyond Pairs and Beyond Examplescited by 4
7 more
  1. 2022LiDAR dataset distillation within bayesian active learning framework: Understanding the effect of data augmentationcited by 2
  2. 2023Multi-Symmetry Ensembles: Improving Diversity and Generalization via Opposing Symmetriescited by 1
  3. 2024Remembering Transformer for Continual Learningcited by 1
  4. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guaranteescited by 1
  5. 2022An Optimal Likelihood Free Method for Biological Model Selectioncited by 0
  6. 2023Meta Neural Coordinationcited by 0
  7. 2024A Unified Framework for Iris Anti-Spoofing: Introducing Iris Anti-Spoofing Cross-Domain-Testing Protocol and Masked-MoE Methodcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
  2. 2023From Sparse to Soft Mixtures of Experts
  3. 2026Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization
  4. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  5. 2026Robustness of Mixtures of Experts to Feature Noise
  6. 2023Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Experts
  7. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
  8. 2024Mixture of Experts in Image Classification: What's the Sweet Spot?