mlsanityData to 14 Oct 2026

Collaboration of Experts: Achieving 80% Top-1 Accuracy on ImageNet with 100M FLOPs

arXiv 2107.03815 ↗First posted 8 Jul 2021cs.CV cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a Collaboration of Experts (CoE) framework to pool together the expertise of multiple networks towards a common aim. Each expert is an individual network with expertise on a unique portion of the dataset, which enhances the collective capacity. Given a sample, an expert is selected by the delegator, which simultaneously outputs a rough prediction to support early termination. To fulfill this framework, we propose three modules to impel each model to play its role, namely weight generation module (WGM), label generation module (LGM) and variance calculation module (VCM). Our method achieves the state-of-the-art performance on ImageNet, 80.7% top-1 accuracy with 194M FLOPs. Combined with PWLU activation function and CondConv, CoE further achieves the accuracy of 80.0% with only 100M FLOPs for the first time. More importantly, our method is hardware friendly and achieves a 3-6x speedup compared with some existing conditional computation approaches.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 42 of its 54 references that are in the map. First posted on arXiv 8 Jul 2021.

Builds on

42 of 54 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  8. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  10. 2019Searching for MobileNetV3cited by 1,994
  11. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  12. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
30 more
  1. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  2. 2016Deep Networks with Stochastic Depthcited by 979
  3. 2017A Closer Look at Memorization in Deep Networkscited by 942
  4. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  5. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  6. 2019GhostNet: More Features from Cheap Operationscited by 410
  7. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
  8. 2019Dynamic Convolution: Attention over Convolution Kernelscited by 356
  9. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  10. 2020Are we done with ImageNet?cited by 333
  11. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  12. 2019Addressing Failure Prediction by Learning Model Confidencecited by 186
  13. 2020Training independent subnetworks for robust predictioncited by 177
  14. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 172
  15. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 150
  16. 2019Adversarial AutoAugmentcited by 111
  17. 2020Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classificationcited by 100
  18. 2021Dynamic Slimmable Networkcited by 83
  19. 2020Ranking Neural Checkpointscited by 51
  20. 2020WeightNet: Revisiting the Design Space of Weight Networkscited by 50
  21. 2020Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNetscited by 49
  22. 2020DyNet: Dynamic Convolution for Accelerating Convolutional Neural Networkscited by 39
  23. 2020FBNetV3: Joint Architecture-Recipe Search using Predictor Pretrainingcited by 35
  24. 2018BlockQNN: Efficient Block-wise Neural Network Architecture Generationcited by 31
  25. 2018InstaNAS: Instance-aware Neural Architecture Searchcited by 13
  26. 2021Learning specialized activation functions with the Piecewise Linear Unitcited by 7
  27. 2021BasisNet: Two-stage Model Synthesis for Efficient Inferencecited by 7
  28. 2019BETANAS: BalancEd TrAining and selective drop for Neural Architecture Searchcited by 5
  29. 2020AutoBSS: An Efficient Algorithm for Block Stacking Style Searchcited by 4
  30. 2020Dynamic Graph: Learning Instance-aware Connectivity for Neural Networkscited by 1

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 1; 2024: 0; 2025: 1.

5 papers.

  1. 2022Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNscited by 155
  2. 2023Revisiting Single-gated Mixtures of Expertscited by 10
  3. 2022Boosted Dynamic Neural Networkscited by 8
  4. 2021Knowledge accumulating: The general pattern of learningcited by 0
  5. 2025CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answeringcited by 0