# Prototypical Contrastive Learning of Unsupervised Representations

arXiv: https://arxiv.org/abs/2005.04966 · HTML: https://mlsanity.com/paper/2005.04966/

## Key facts

- Cited by 640 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2020.

## Built on by (most cited first, 30 of 640)

- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [Context Autoencoder for Self-Supervised Representation Learning](https://mlsanity.com/paper/2202.03026/) (2022)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Self-supervised Pretraining of Visual Features in the Wild](https://mlsanity.com/paper/2103.01988/) (2021)
- [Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals](https://mlsanity.com/paper/2102.06191/) (2021)
- [BEVT: BERT Pretraining of Video Transformers](https://mlsanity.com/paper/2112.01529/) (2021)
- [TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series](https://mlsanity.com/paper/2308.08241/) (2023)
- [On Feature Decorrelation in Self-Supervised Learning](https://mlsanity.com/paper/2105.00470/) (2021)
- [Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning](https://mlsanity.com/paper/2210.06044/) (2022)
- [Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization](https://mlsanity.com/paper/2205.07839/) (2022)
- [Efficient Self-supervised Vision Transformers for Representation Learning](https://mlsanity.com/paper/2106.09785/) (2021)
- [Balanced Contrastive Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2207.09052/) (2022)
- [Rethinking Semantic Segmentation: A Prototype View](https://mlsanity.com/paper/2203.15102/) (2022)
- [Self-supervised Learning in Remote Sensing: A Review](https://mlsanity.com/paper/2206.13188/) (2022)
- [SEED: Self-supervised Distillation For Visual Representation](https://mlsanity.com/paper/2101.04731/) (2021)
- [How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?](https://mlsanity.com/paper/2203.04450/) (2022)
- [Self-supervised Graph-level Representation Learning with Local and Global Structure](https://mlsanity.com/paper/2106.04113/) (2021)
- [SimMatch: Semi-supervised Learning with Similarity Matching](https://mlsanity.com/paper/2203.06915/) (2022)
- [What to Hide from Your Students: Attention-Guided Masked Image Modeling](https://mlsanity.com/paper/2203.12719/) (2022)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Large-scale Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2106.03149/) (2021)
- [Contrastive Learning with Stronger Augmentations](https://mlsanity.com/paper/2104.07713/) (2021)
- [SiT: Self-supervised vIsion Transformer](https://mlsanity.com/paper/2104.03602/) (2021)

## Abstract (arXiv)

This paper presents Prototypical Contrastive Learning (PCL), an unsupervised representation learning method that addresses the fundamental limitations of instance-wise contrastive learning. PCL not only learns low-level features for the task of instance discrimination, but more importantly, it implicitly encodes semantic structures of the data into the learned embedding space. Specifically, we introduce prototypes as latent variables to help find the maximum-likelihood estimation of the network parameters in an Expectation-Maximization framework. We iteratively perform E-step as finding the distribution of prototypes via clustering and M-step as optimizing the network via contrastive learning. We propose ProtoNCE loss, a generalized version of the InfoNCE loss for contrastive learning, which encourages representations to be closer to their assigned prototypes. PCL outperforms state-of-the-art instance-wise contrastive learning methods on multiple benchmarks with substantial improvement in low-resource transfer learning. Code and pretrained models are available at https://github.com/salesforce/PCL.