# Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning

arXiv: https://arxiv.org/abs/2011.10043 · HTML: https://mlsanity.com/paper/2011.10043/

## Key facts

- Cited by 297 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2020.

## Built on by (most cited first, 30 of 297)

- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [Context Autoencoder for Self-Supervised Representation Learning](https://mlsanity.com/paper/2202.03026/) (2022)
- [SpectralGPT: Spectral Remote Sensing Foundation Model](https://mlsanity.com/paper/2311.07113/) (2023)
- [Exploring Cross-Image Pixel Contrast for Semantic Segmentation](https://mlsanity.com/paper/2101.11939/) (2021)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything](https://mlsanity.com/paper/2312.00863/) (2023)
- [CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow](https://mlsanity.com/paper/2211.10408/) (2022)
- [CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion](https://mlsanity.com/paper/2210.10716/) (2022)
- [Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning](https://mlsanity.com/paper/2210.06044/) (2022)
- [Efficient Visual Pretraining with Contrastive Detection](https://mlsanity.com/paper/2103.10957/) (2021)
- [Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization](https://mlsanity.com/paper/2205.07839/) (2022)
- [Efficient Self-supervised Vision Transformers for Representation Learning](https://mlsanity.com/paper/2106.09785/) (2021)
- [Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data](https://mlsanity.com/paper/2203.16258/) (2022)
- [Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bank](https://mlsanity.com/paper/2104.13415/) (2021)
- [Segment Any Point Cloud Sequences by Distilling Vision Foundation Models](https://mlsanity.com/paper/2306.09347/) (2023)
- [Self-Supervised Learning with Swin Transformers](https://mlsanity.com/paper/2105.04553/) (2021)
- [VICRegL: Self-Supervised Learning of Local Visual Features](https://mlsanity.com/paper/2210.01571/) (2022)
- [Revealing the Dark Secrets of Masked Image Modeling](https://mlsanity.com/paper/2205.13543/) (2022)
- [MST: Masked Self-Supervised Transformer for Visual Representation](https://mlsanity.com/paper/2106.05656/) (2021)
- [An Empirical Study of Graph Contrastive Learning](https://mlsanity.com/paper/2109.01116/) (2021)
- [Aligning Pretraining for Detection via Object-Level Contrastive Learning](https://mlsanity.com/paper/2106.02637/) (2021)
- [FreeSOLO: Learning to Segment Objects without Annotations](https://mlsanity.com/paper/2202.12181/) (2022)
- [Large-scale Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2106.03149/) (2021)
- [Pixel Contrastive-Consistent Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2108.09025/) (2021)
- [Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation](https://mlsanity.com/paper/2205.14141/) (2022)
- [SePiCo: Semantic-Guided Pixel Contrast for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2204.08808/) (2022)
- [Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training](https://mlsanity.com/paper/2308.09718/) (2023)
- [Joint Learning of Localized Representations from Medical Images and Reports](https://mlsanity.com/paper/2112.02889/) (2021)

## Abstract (arXiv)

Contrastive learning methods for unsupervised visual representation learning have reached remarkable levels of transfer performance. We argue that the power of contrastive learning has yet to be fully unleashed, as current methods are trained only on instance-level pretext tasks, leading to representations that may be sub-optimal for downstream tasks requiring dense pixel predictions. In this paper, we introduce pixel-level pretext tasks for learning dense feature representations. The first task directly applies contrastive learning at the pixel level. We additionally propose a pixel-to-propagation consistency task that produces better results, even surpassing the state-of-the-art approaches by a large margin. Specifically, it achieves 60.2 AP, 41.4 / 40.5 mAP and 77.2 mIoU when transferred to Pascal VOC object detection (C4), COCO object detection (FPN / C4) and Cityscapes semantic segmentation using a ResNet-50 backbone network, which are 2.6 AP, 0.8 / 1.0 mAP and 1.0 mIoU better than the previous best methods built on instance-level contrastive learning. Moreover, the pixel-level pretext tasks are found to be effective for pre-training not only regular backbone networks but also head networks used for dense downstream tasks, and are complementary to instance-level contrastive methods. These results demonstrate the strong potential of defining pretext tasks at the pixel level, and suggest a new path forward in unsupervised visual representation learning. Code is available at urlhttps://github.com/zdaxie/PixPro.