mlsanityData to 7 Oct 2026

Exploring the Limits of Weakly Supervised Pretraining

arXiv 1805.00932 ↗First posted 2 May 2018cs.CVcited by 604

Not on the map. Why?

Abstract (arXiv)

State-of-the-art visual perception models for a wide range of tasks rely on supervised pretraining. ImageNet classification is the de facto pretraining task for these models. Yet, ImageNet is now nearly ten years old and is by modern standards "small". Even so, relatively little is known about the behavior of pretraining with datasets that are multiple orders of magnitude larger. The reasons are obvious: such datasets are difficult to collect and annotate. In this paper, we present a unique study of transfer learning with large convolutional networks trained to predict hashtags on billions of social media images. Our experiments demonstrate that training for large-scale hashtag prediction leads to excellent results. We show improvements on several image classification and object detection tasks, and report the highest ImageNet-1k single-crop, top-1 accuracy to date: 85.4% (97.6% top-5). We also perform extensive experiments that provide novel empirical data on the relationship between large-scale pretraining and transfer learning performance.

Key facts. Cited by 604 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 185; 2022: 162; 2023: 122; 2024: 67; 2025: 46; 2026: 20.

604 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2021Vision Transformers for Dense Predictioncited by 1,789
  8. 2022Visual Prompt Tuningcited by 1,674
  9. 2021ViViT: A Video Vision Transformercited by 1,610
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
38 more
  1. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  4. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  5. 2021Scaling Vision Transformerscited by 993
  6. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  7. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  8. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  9. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  10. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  11. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  12. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  13. 2021Deep Long-Tailed Learning: A Surveycited by 416
  14. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  15. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  16. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  17. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 376
  18. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  19. 2021Domain Adaptation for Medical Image Analysis: A Surveycited by 351
  20. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  21. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  22. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  23. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  24. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  25. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  26. 2021Scaling Laws for Transfercited by 265
  27. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  28. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  29. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  30. 2021Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Datacited by 214
  31. 2021Accelerating Sparse Deep Neural Networkscited by 213
  32. 2021Vision Transformers are Robust Learnerscited by 208
  33. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  34. 2021Distribution Alignment: A Unified Framework for Long-tail Visual Recognitioncited by 198
  35. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  36. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  37. 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 193
  38. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182

50 most cited of 604 All 604 (JSON)