mlsanityData to 14 Oct 2026

Momentum Contrast for Unsupervised Visual Representation Learning

arXiv 1911.05722 ↗First posted 13 Nov 2019cs.CVcited by 7,513

Not on the map. Why?

Cited by 203 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dictionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks. MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large margins. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.

Key facts. Cited by 7,513 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1262; 2022: 1660; 2023: 1696; 2024: 1298; 2025: 1000; 2026: 597.

7,513 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  11. 2023Consistency Modelscited by 1,822
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
38 more
  1. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  2. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  3. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  5. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  6. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  7. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  8. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  9. 2021Transformers in Vision: A Surveycited by 1,013
  10. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  11. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  12. 2023Vision Transformers Need Registerscited by 871
  13. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  14. 2022A Generalist Agentcited by 808
  15. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  16. 2021Model-Contrastive Federated Learningcited by 766
  17. 2021Domain Generalization: A Surveycited by 764
  18. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  19. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  20. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  21. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 712
  22. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  23. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  24. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  25. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  26. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  27. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  28. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  29. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  30. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  31. 2021Extract Free Dense Labels from CLIPcited by 583
  32. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  33. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  34. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  35. 2021CutPaste: Self-Supervised Learning for Anomaly Detection and Localizationcited by 531
  36. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  37. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  38. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459

50 most cited of 7,513 All 7,513 (JSON)