mlsanityData to 14 Oct 2026

Revisiting Weakly Supervised Pre-Training of Visual Perception Models

arXiv 2201.08371 ↗First posted 20 Jan 2022cs.CVcited by 116

Not on the map. Why?

Abstract (arXiv)

Model pre-training is a cornerstone of modern visual recognition systems. Although fully supervised pre-training on datasets like ImageNet is still the de-facto standard, recent studies suggest that large-scale weakly supervised pre-training can outperform fully supervised approaches. This paper revisits weakly-supervised pre-training of models using hashtag supervision with modern versions of residual networks and the largest-ever dataset of images and corresponding hashtags. We study the performance of the resulting models in various transfer-learning settings including zero-shot transfer. We also compare our models with those obtained via large-scale self-supervised learning. We find our weakly-supervised models to be very competitive across all settings, and find they substantially outperform their self-supervised counterparts. We also include an investigation into whether our models learned potentially troubling associations or stereotypes. Overall, our results provide a compelling argument for the use of weakly supervised learning in the development of visual recognition systems. Our models, Supervised Weakly through hashtAGs (SWAG), are available publicly.

Key facts. Cited by 116 papers in the map (data through 14 Oct 2026). Builds on 60 of its 80 references that are in the map. First posted on arXiv 20 Jan 2022.

Builds on

60 of 80 references.

References · 60

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2017Mask R-CNNcited by 7,277
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2016Densely Connected Convolutional Networkscited by 6,720
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  12. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
38 more
  1. 2017Squeeze-and-Excitation Networkscited by 4,789
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  7. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  9. 2017The Kinetics Human Action Video Datasetcited by 1,909
  10. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  12. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  13. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  14. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  15. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  16. 2021Scaling Vision Transformerscited by 999
  17. 2019When Does Label Smoothing Help?cited by 988
  18. 2016Deep Networks with Stochastic Depthcited by 979
  19. 2020Designing Network Design Spacescited by 960
  20. 2021Multiscale Vision Transformerscited by 897
  21. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  22. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  23. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  24. 2021ImageNet-21K Pretraining for the Massescited by 613
  25. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  26. 2018Do Better ImageNet Models Transfer Better?cited by 565
  27. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 429
  28. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
  29. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  30. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
  31. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  32. 2019Adversarial Examples Improve Image Recognitioncited by 286
  33. 2019Fixing the train-test resolution discrepancycited by 209
  34. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 198
  35. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  36. 2019Does Object Recognition Work for Everyone?cited by 151
  37. 2021TransFG: A Transformer Architecture for Fine-grained Recognitioncited by 132
  38. 2019Large-scale weakly-supervised pre-training for video action recognitioncited by 130

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 18; 2023: 44; 2024: 33; 2025: 16; 2026: 5.

116 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  2. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  3. 2022Token Merging: Your ViT But Fastercited by 1,021
  4. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  5. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  6. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 193
  7. 2022Domain Generalization by Mutual-Information Regularization with Pre-trained Modelscited by 133
  8. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  9. 2022A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Otherscited by 85
  10. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  11. 2023The effectiveness of MAE pre-pretraining for billion-scale pretrainingcited by 68
  12. 2024IMPRINT: Generative Object Compositing by Learning Identity-Preserving Representationcited by 64
38 more
  1. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 62
  2. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  3. 2023Very high resolution canopy height maps from RGB imagery using self-supervised vision transformer and convolutional decoder trained on Aerial Lidarcited by 60
  4. 2023FACET: Fairness in Computer Vision Evaluation Benchmarkcited by 58
  5. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustnesscited by 48
  6. 2022ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCOcited by 37
  7. 2023Vision Transformers with Mixed-Resolution Tokenizationcited by 36
  8. 2023Unicom: Universal and Compact Representation Learning for Image Retrievalcited by 36
  9. 2024AMEGO: Active Memory from long EGOcentric videoscited by 36
  10. 2023What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifierscited by 35
  11. 2023Visual Tuningcited by 34
  12. 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33
  13. 2022Towards All-in-one Pre-training via Maximizing Multi-modal Mutual Informationcited by 31
  14. 2023A framework for benchmarking class-out-of-distribution detection and its application to ImageNetcited by 31
  15. 2023CiT: Curation in Training for Effective Vision-Language Datacited by 27
  16. 2022Towards a Unified View on Visual Parameter-Efficient Transfer Learningcited by 25
  17. 2022CAE v2: Context Autoencoder with CLIP Targetcited by 25
  18. 2022On the Strong Correlation Between Model Invariance and Generalizationcited by 23
  19. 2024Rethinking Multi-domain Generalization with A General Learning Objectivecited by 23
  20. 2024MIM-Refiner: A Contrastive Learning Boost from Intermediate Pre-Trained Representationscited by 19
  21. 2025Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuningcited by 18
  22. 2022You Only Need a Good Embeddings Extractor to Fix Spurious Correlationscited by 17
  23. 2023GeoNet: Benchmarking Unsupervised Adaptation across Geographiescited by 17
  24. 2024Towards Geographic Inclusion in the Evaluation of Text-to-Image Modelscited by 16
  25. 2023Brain Decodes Deep Netscited by 14
  26. 2024Let Go of Your Labels with Unsupervised Transfercited by 14
  27. 2024Multi-label Cluster Discrimination for Visual Representation Learningcited by 14
  28. 2022Progress and limitations of deep networks to recognize objects in unusual posescited by 13
  29. 2023Effective Robustness against Natural Distribution Shifts for Models with Different Training Datacited by 13
  30. 2023Adversarial Attacks on Foundational Vision Modelscited by 13
  31. 2024Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Large Modelscited by 13
  32. 2024Agglomerative Token Clusteringcited by 13
  33. 2023Domain Generalization Using Large Pretrained Models with Mixture-of-Adapterscited by 11
  34. 2023Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Modelscited by 11
  35. 2024Feature contamination: Neural networks learn uncorrelated features and fail to generalizecited by 11
  36. 2023Are Deep Neural Networks Adequate Behavioural Models of Human Visual Perception?cited by 10
  37. 2023Contextual Vision Transformers for Robust Representation Learningcited by 10
  38. 2023Class-Discriminative Attention Maps for Vision Transformerscited by 10

50 most cited of 116 All 116 (JSON)