mlsanityData to 7 Oct 2026

ImageNet-21K Pretraining for the Masses

arXiv 2104.10972 ↗First posted 22 Apr 2021cs.CV cs.LGcited by 607

Not on the map. Why?

Cited by 44 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

ImageNet-1K serves as the primary dataset for pretraining deep learning models for computer vision tasks. ImageNet-21K dataset, which is bigger and more diverse, is used less frequently for pretraining, mainly due to its complexity, low accessibility, and underestimation of its added value. This paper aims to close this gap, and make high-quality efficient pretraining on ImageNet-21K available for everyone. Via a dedicated preprocessing stage, utilization of WordNet hierarchical structure, and a novel training scheme called semantic softmax, we show that various models significantly benefit from ImageNet-21K pretraining on numerous datasets and tasks, including small mobile-oriented models. We also show that we outperform previous ImageNet-21K pretraining schemes for prominent new models like ViT and Mixer. Our proposed pretraining pipeline is efficient, accessible, and leads to SoTA reproducible results, from a publicly available dataset. The training code and pretrained models are available at: https://github.com/Alibaba-MIIL/ImageNet21K

Key facts. Cited by 607 papers in the map (data through 7 Oct 2026). Builds on 49 of its 62 references that are in the map. First posted on arXiv 22 Apr 2021.

Builds on

49 of 62 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  8. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  9. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  10. 2019Searching for MobileNetV3cited by 1,984
  11. 2017The Kinetics Human Action Video Datasetcited by 1,894
  12. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
37 more
  1. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  2. 2020Knowledge Distillation: A Surveycited by 1,578
  3. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  4. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  5. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  6. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  7. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  8. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  9. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  10. 2018Do Better ImageNet Models Transfer Better?cited by 560
  11. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  12. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
  13. 2013Do Deep Nets Really Need to be Deep?cited by 466
  14. 2020Asymmetric Loss For Multi-Label Classificationcited by 379
  15. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 374
  16. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  17. 2019A Baseline for Few-Shot Image Classificationcited by 298
  18. 2021Video Transformer Networkcited by 271
  19. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  20. 2018A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decaycited by 223
  21. 2020Understanding the Difficulty of Training Transformerscited by 210
  22. 2016What makes ImageNet good for transfer learning?cited by 192
  23. 2020fastai: A Layered API for Deep Learningcited by 183
  24. 2020TResNet: High Performance GPU-Dedicated Architecturecited by 119
  25. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 117
  26. 2018Adapting Auxiliary Losses Using Gradient Similaritycited by 98
  27. 2021An Image is Worth 16x16 Words, What is a Video Worth?cited by 87
  28. 2019ClusterFit: Improving Generalization of Visual Representationscited by 82
  29. 2021CPTR: Full Transformer Network for Image Captioningcited by 55
  30. 2020Scalable Transfer Learning with Expert Modelscited by 48
  31. 2019FoodX-251: A Dataset for Fine-grained Food Classificationcited by 33
  32. 2018Domain Adaptive Transfer Learning with Specialist Modelscited by 31
  33. 2018Integrating domain knowledge: using hierarchies to improve deep classifierscited by 25
  34. 2020Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Networkcited by 24
  35. 2020Deep Ensembles for Low-Data Transfer Learningcited by 19
  36. 2017Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to traincited by 8
  37. 2014Deep Networks With Large Output Spacescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 80; 2023: 141; 2024: 164; 2025: 115; 2026: 77.

607 papers.

Sort:

By area:

  1. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  2. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  3. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  4. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  5. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  6. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  7. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  8. 2021Early Convolutions Help Transformers See Bettercited by 395
  9. 2021Open-Set Recognition: a Good Closed-Set Classifier is All You Need?cited by 351
  10. 2022Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Frameworkcited by 328
  11. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  12. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
38 more
  1. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  2. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  3. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  4. 2022Parameter-free Online Test-time Adaptationcited by 184
  5. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  6. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 178
  7. 2023OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detectioncited by 178
  8. 2021ResViT: Residual vision transformers for multi-modal medical image synthesiscited by 177
  9. 2022Separable Self-attention for Mobile Vision Transformerscited by 163
  10. 2023MedNeXt: Transformer-driven Scaling of ConvNets for Medical Image Segmentationcited by 161
  11. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  12. 2021Query2Label: A Simple Transformer Way to Multi-Label Classificationcited by 126
  13. 2022EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applicationscited by 121
  14. 2023Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimalitycited by 121
  15. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  16. 2023Change is Hard: A Closer Look at Subpopulation Shiftcited by 119
  17. 20234M: Massively Multimodal Masked Modelingcited by 118
  18. 2022Revisiting Weakly Supervised Pre-Training of Visual Perception Modelscited by 116
  19. 2021Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learningcited by 112
  20. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  21. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  22. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  23. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
  24. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
  25. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 105
  26. 2022HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutionscited by 103
  27. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  28. 2022Self-Supervised Learning of Object Parts for Semantic Segmentationcited by 90
  29. 2023FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learningcited by 86
  30. 2024All in One and One for All: A Simple yet Effective Method towards Cross-domain Graph Pretrainingcited by 84
  31. 2021ML-Decoder: Scalable and Versatile Classification Headcited by 78
  32. 2022Deep Model Reassemblycited by 78
  33. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  34. 2023The effectiveness of MAE pre-pretraining for billion-scale pretrainingcited by 68
  35. 2024Parameter-Efficient Fine-Tuning with Discrete Fourier Transformcited by 67
  36. 2023Token Contrast for Weakly-Supervised Semantic Segmentationcited by 64
  37. 2023PILOT: A Pre-Trained Model-Based Continual Learning Toolboxcited by 64
  38. 2022Video Transformers: A Surveycited by 63

50 most cited of 607 All 607 (JSON)