mlsanityData to 14 Oct 2026

Dense Contrastive Learning for Self-Supervised Visual Pre-Training

arXiv 2011.09157 ↗First posted 18 Nov 2020cs.CVcited by 512

Not on the map. Why?

Abstract (arXiv)

To date, most existing self-supervised learning methods are designed and optimized for image classification. These pre-trained models can be sub-optimal for dense prediction tasks due to the discrepancy between image-level prediction and pixel-level prediction. To fill this gap, we aim to design an effective, dense self-supervised learning method that directly works at the level of pixels (or local features) by taking into account the correspondence between local features. We present dense contrastive learning, which implements self-supervised learning by optimizing a pairwise contrastive (dis)similarity loss at the pixel level between two views of input images. Compared to the baseline method MoCo-v2, our method introduces negligible computation overhead (only <1% slower), but demonstrates consistently superior performance when transferring to downstream dense prediction tasks including object detection, semantic segmentation and instance segmentation; and outperforms the state-of-the-art methods by a large margin. Specifically, over the strong MoCo-v2 baseline, our method achieves significant improvements of 2.0% AP on PASCAL VOC object detection, 1.1% AP on COCO object detection, 0.9% AP on COCO instance segmentation, 3.0% mIoU on PASCAL VOC semantic segmentation and 1.8% mIoU on Cityscapes semantic segmentation. Code is available at: https://git.io/AdelaiDet

Key facts. Cited by 512 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 58; 2022: 161; 2023: 137; 2024: 70; 2025: 62; 2026: 24.

512 papers.

Sort:

By area:

  1. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  2. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  3. 2021TS2Vec: Towards Universal Representation of Time Seriescited by 471
  4. 2022SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentationcited by 400
  5. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  6. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  7. 2023A Cookbook of Self-Supervised Learningcited by 250
  8. 2021Exploring Cross-Image Pixel Contrast for Semantic Segmentationcited by 242
  9. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  10. 2021DetCo: Unsupervised Contrastive Learning for Object Detectioncited by 205
  11. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 188
  12. 2023Cut and Learn for Unsupervised Object Detection and Instance Segmentationcited by 187
38 more
  1. 2023CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoderscited by 169
  2. 2022CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flowcited by 166
  3. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 160
  4. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  5. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  6. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  7. 2022Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detectioncited by 130
  8. 2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
  9. 2021Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bankcited by 125
  10. 2023Segment Any Point Cloud Sequences by Distilling Vision Foundation Modelscited by 125
  11. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  12. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  13. 2021MST: Masked Self-Supervised Transformer for Visual Representationcited by 114
  14. 2021Aligning Pretraining for Detection via Object-Level Contrastive Learningcited by 110
  15. 2022FreeSOLO: Learning to Segment Objects without Annotationscited by 102
  16. 2021Large-scale Unsupervised Semantic Segmentationcited by 101
  17. 2021Pixel Contrastive-Consistent Semi-Supervised Semantic Segmentationcited by 101
  18. 2022BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotationscited by 99
  19. 2021Region Similarity Representation Learningcited by 96
  20. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
  21. 2023What Do Self-Supervised Vision Transformers Learn?cited by 93
  22. 2022Self-Supervised Learning of Object Parts for Semantic Segmentationcited by 90
  23. 2022SePiCo: Semantic-Guided Pixel Contrast for Domain Adaptive Semantic Segmentationcited by 87
  24. 2023Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Trainingcited by 86
  25. 2021Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrastcited by 85
  26. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  27. 2021Bootstrapping Semantic Segmentation with Regional Contrastcited by 80
  28. 2022Siamese Image Modeling for Self-Supervised Vision Representation Learningcited by 78
  29. 2021DETReg: Unsupervised Pretraining with Region Priors for Object Detectioncited by 76
  30. 2022CORE: Consistent Representation Learning for Face Forgery Detectioncited by 74
  31. 2021Unsupervised Object-Level Representation Learning from Scene Imagescited by 72
  32. 2022Self-Supervised Visual Representation Learning with Semantic Groupingcited by 70
  33. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  34. 2022Crafting Better Contrastive Views for Siamese Representation Learningcited by 67
  35. 2022Unsupervised Object Localization: Observing the Background to Discover Objectscited by 66
  36. 2022Regional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentationcited by 64
  37. 2022Hyperbolic Contrastive Learning for Visual Representations beyond Objectscited by 61
  38. 2022Patch-level Representation Learning for Self-supervised Vision Transformerscited by 60

50 most cited of 512 All 512 (JSON)