mlsanityData to 7 Oct 2026

Big Self-Supervised Models are Strong Semi-Supervised Learners

arXiv 2006.10029 ↗First posted 17 Jun 2020cs.LG cs.CV stat.MLcited by 1,375

Not on the map. Why?

Cited by 26 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

One paradigm for learning from few labeled examples while making best use of a large amount of unlabeled data is unsupervised pretraining followed by supervised fine-tuning. Although this paradigm uses unlabeled data in a task-agnostic way, in contrast to common approaches to semi-supervised learning for computer vision, we show that it is surprisingly effective for semi-supervised learning on ImageNet. A key ingredient of our approach is the use of big (deep and wide) networks during pretraining and fine-tuning. We find that, the fewer the labels, the more this approach (task-agnostic use of unlabeled data) benefits from a bigger network. After fine-tuning, the big network can be further improved and distilled into a much smaller one with little loss in classification accuracy by using the unlabeled examples for a second time, but in a task-specific way. The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2, supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge. This procedure achieves 73.9% ImageNet top-1 accuracy with just 1% of the labels (≤13 labeled images per class) using ResNet-50, a 10× improvement in label efficiency over the previous state-of-the-art. With 10% of labels, ResNet-50 trained with our method achieves 77.5% top-1 accuracy, outperforming standard supervised training with all of the labels.

Key facts. Cited by 1,375 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 353; 2022: 358; 2023: 316; 2024: 196; 2025: 102; 2026: 50.

1,375 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  4. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  6. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  7. 2021Scaling Vision Transformerscited by 993
  8. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  9. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  10. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  11. 2021Robust fine-tuning of zero-shot modelscited by 785
  12. 2021Model-Contrastive Federated Learningcited by 762
38 more
  1. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  2. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  3. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  5. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  6. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  7. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  8. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  9. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  10. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  11. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  12. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  13. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  14. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  15. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  16. 2021Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentationcited by 268
  17. 2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
  18. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 262
  19. 2022Contrastive Test-Time Adaptationcited by 258
  20. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  21. 2021A Survey on Deep Semi-supervised Learningcited by 250
  22. 2023SpectralGPT: Spectral Remote Sensing Foundation Modelcited by 250
  23. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  24. 2023A Cookbook of Self-Supervised Learningcited by 248
  25. 2022Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistencycited by 246
  26. 2022Transformers in Medical Imaging: A Surveycited by 245
  27. 2022Unsupervised Semantic Segmentation by Distilling Feature Correspondencescited by 241
  28. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  29. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  30. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  31. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  32. 2022Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labelscited by 218
  33. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  34. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  35. 2021Vision Transformers are Robust Learnerscited by 208
  36. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  37. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  38. 2021Adversarial Graph Augmentation to Improve Graph Contrastive Learningcited by 200

50 most cited of 1,375 All 1,375 (JSON)