mlsanityData to 13 Oct 2026

A Simple Framework for Contrastive Learning of Visual Representations

arXiv 2002.05709 ↗First posted 13 Feb 2020cs.LG cs.CV stat.MLcited by 12,009

Not on the map. Why?

Cited by 291 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.

Key facts. Cited by 12,009 papers in the map (data through 13 Oct 2026). First posted on arXiv 13 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2020: 2; 2021: 1682; 2022: 2360; 2023: 2561; 2024: 2198; 2025: 1890; 2026: 1316.

12,009 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,857
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,068
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,368
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,276
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,170
  11. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,826
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,616
38 more
  1. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,612
  2. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,431
  3. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,229
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,227
  5. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,205
  6. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,168
  7. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,150
  8. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,033
  9. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,018
  10. 2022Diffusion Models in Vision: A Surveycited by 1,012
  11. 2021Transformers in Vision: A Surveycited by 1,010
  12. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 895
  13. 2023Vision-Language Models for Vision Tasks: A Surveycited by 837
  14. 2021Florence: A New Foundation Model for Computer Visioncited by 803
  15. 2021Robust fine-tuning of zero-shot modelscited by 790
  16. 2021Model-Contrastive Federated Learningcited by 766
  17. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 744
  18. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 732
  19. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 723
  20. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 721
  21. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 705
  22. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 704
  23. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 692
  24. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 679
  25. 2021Vector-quantized Image Modeling with Improved VQGANcited by 657
  26. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  27. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 615
  28. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 606
  29. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 600
  30. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 598
  31. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 594
  32. 2021Extract Free Dense Labels from CLIPcited by 582
  33. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 574
  34. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 573
  35. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 555
  36. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  37. 2021CutPaste: Self-Supervised Learning for Anomaly Detection and Localizationcited by 530
  38. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 520

50 most cited of 12,009 All 12,009 (JSON)