mlsanityData to 7 Oct 2026

Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results

arXiv 2204.03475 ↗First posted 7 Apr 2022cs.CV cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

ImageNet serves as the primary dataset for evaluating the quality of computer-vision models. The common practice today is training each architecture with a tailor-made scheme, designed and tuned by an expert. In this paper, we present a unified scheme for training any backbone on ImageNet. The scheme, named USI (Unified Scheme for ImageNet), is based on knowledge distillation and modern tricks. It requires no adjustments or hyper-parameters tuning between different models, and is efficient in terms of training times. We test USI on a wide variety of architectures, including CNNs, Transformers, Mobile-oriented and MLP-only. On all models tested, USI outperforms previous state-of-the-art results. Hence, we are able to transform training on ImageNet from an expert-oriented task to an automatic seamless routine. Since USI accepts any backbone and trains it to top results, it also enables to perform methodical comparisons, and identify the most efficient backbones along the speed-accuracy Pareto curve. Implementation is available at:https://github.com/Alibaba-MIIL/Solving_ImageNet

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 42 of its 47 references that are in the map. First posted on arXiv 7 Apr 2022.

Builds on

42 of 47 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  8. 2017Squeeze-and-Excitation Networkscited by 4,763
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2022A ConvNet for the 2020scited by 3,545
  11. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
30 more
  1. 2019Searching for MobileNetV3cited by 1,984
  2. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  3. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  5. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  6. 2019When Does Label Smoothing Help?cited by 985
  7. 2020Designing Network Design Spacescited by 954
  8. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  9. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  10. 2021Transformer in Transformercited by 632
  11. 2021ImageNet-21K Pretraining for the Massescited by 607
  12. 2018Do Better ImageNet Models Transfer Better?cited by 560
  13. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  14. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  15. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  16. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  17. 2018DropBlock: A regularization method for convolutional networkscited by 261
  18. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  19. 2020Understanding the Difficulty of Training Transformerscited by 210
  20. 2019Fixing the train-test resolution discrepancycited by 209
  21. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 196
  22. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 164
  23. 2020TResNet: High Performance GPU-Dedicated Architecturecited by 119
  24. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 117
  25. 2018A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillationcited by 110
  26. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  27. 2019MultiGrain: a unified image embedding for classes and instancescited by 85
  28. 2020Circumventing Outliers of AutoAugment with Knowledge Distillationcited by 29
  29. 2020Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Networkcited by 24
  30. 2019Effective and Efficient Dropout for Deep Convolutional Neural Networkscited by 21

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 1; 2024: 1; 2025: 1.

8 papers.

  1. 2022EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applicationscited by 121
  2. 2022Join the High Accuracy Club on ImageNet with A Binary Neural Network Ticketcited by 13
  3. 2023Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcementcited by 11
  4. 2022Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASRcited by 10
  5. 2024DailyMAE: Towards Pretraining Masked Autoencoders in One Daycited by 2
  6. 2025A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategycited by 1
  7. 2022Generalized Supervised Contrastive Learningcited by 0
  8. 2022Effectiveness of Function Matching in Driving Scene Recognitioncited by 0