mlsanityData to 7 Oct 2026

Improved Baselines with Momentum Contrastive Learning

arXiv 2003.04297 ↗First posted 9 Mar 2020cs.CVcited by 2,282

Not on the map. Why?

Cited by 43 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive unsupervised learning has recently shown encouraging progress, e.g., in Momentum Contrast (MoCo) and SimCLR. In this note, we verify the effectiveness of two of SimCLR's design improvements by implementing them in the MoCo framework. With simple modifications to MoCo---namely, using an MLP projection head and more data augmentation---we establish stronger baselines that outperform SimCLR and do not require large training batches. We hope this will make state-of-the-art unsupervised learning research more accessible. Code will be made public.

Key facts. Cited by 2,282 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 465; 2022: 567; 2023: 545; 2024: 392; 2025: 207; 2026: 106.

2,282 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  6. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  8. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  9. 2021Transformers in Vision: A Surveycited by 1,009
  10. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  11. 2021Robust fine-tuning of zero-shot modelscited by 785
  12. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
38 more
  1. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  2. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  3. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  4. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  5. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 542
  6. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  7. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  8. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  9. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  10. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  11. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  12. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  13. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  14. 2021Open-Set Recognition: a Good Closed-Set Classifier is All You Need?cited by 351
  15. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  16. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  17. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  18. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 320
  19. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  20. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  21. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
  22. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  23. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  24. 2022Contrastive Test-Time Adaptationcited by 258
  25. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  26. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  27. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  28. 2023SpectralGPT: Spectral Remote Sensing Foundation Modelcited by 250
  29. 2022Visual Prompting via Image Inpaintingcited by 248
  30. 2023A Cookbook of Self-Supervised Learningcited by 248
  31. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  32. 2021Exploring Cross-Image Pixel Contrast for Semantic Segmentationcited by 242
  33. 2022Unsupervised Semantic Segmentation by Distilling Feature Correspondencescited by 241
  34. 2023MVImgNet: A Large-scale Dataset of Multi-view Imagescited by 219
  35. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  36. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  37. 2021Parametric Contrastive Learningcited by 217
  38. 2021Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Datacited by 214

50 most cited of 2,282 All 2,282 (JSON)