# With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations

arXiv: https://arxiv.org/abs/2104.14548 · HTML: https://mlsanity.com/paper/2104.14548/

## Key facts

- Cited by 326 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 77 references that are in the map.
- First posted on arXiv 29 Apr 2021.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)

## Built on by (most cited first, 30 of 326)

- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation](https://mlsanity.com/paper/2110.04202/) (2021)
- [The Surprising Effectiveness of Representation Learning for Visual Imitation](https://mlsanity.com/paper/2112.01511/) (2021)
- [Self-Supervised Models are Continual Learners](https://mlsanity.com/paper/2112.04215/) (2021)
- [Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods](https://mlsanity.com/paper/2205.11508/) (2022)
- [Decoupled Contrastive Learning](https://mlsanity.com/paper/2110.06848/) (2021)
- [SimMatch: Semi-supervised Learning with Similarity Matching](https://mlsanity.com/paper/2203.06915/) (2022)
- [VICRegL: Self-Supervised Learning of Local Visual Features](https://mlsanity.com/paper/2210.01571/) (2022)
- [Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning](https://mlsanity.com/paper/2108.01775/) (2021)
- [Efficient Training of Visual Transformers with Small Datasets](https://mlsanity.com/paper/2106.03746/) (2021)
- [Attracting and Dispersing: A Simple Approach for Source-free Domain Adaptation](https://mlsanity.com/paper/2205.04183/) (2022)
- [Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition](https://mlsanity.com/paper/2112.03590/) (2021)
- [PromptCAL: Contrastive Affinity Learning via Auxiliary Prompts for Generalized Novel Category Discovery](https://mlsanity.com/paper/2212.05590/) (2022)
- [Object discovery and representation networks](https://mlsanity.com/paper/2203.08777/) (2022)
- [WildNet: Learning Domain Generalized Semantic Segmentation from the Wild](https://mlsanity.com/paper/2204.01446/) (2022)
- [Unsupervised Object-Level Representation Learning from Scene Images](https://mlsanity.com/paper/2106.11952/) (2021)
- [Self-Supervised Visual Representation Learning with Semantic Grouping](https://mlsanity.com/paper/2205.15288/) (2022)
- [Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?](https://mlsanity.com/paper/2201.05119/) (2022)
- [Crafting Better Contrastive Views for Siamese Representation Learning](https://mlsanity.com/paper/2202.03278/) (2022)
- [Assessing the State of Self-Supervised Human Activity Recognition using Wearables](https://mlsanity.com/paper/2202.12938/) (2022)
- [Test-Time Adaptation via Self-Training with Nearest Neighbor Information](https://mlsanity.com/paper/2207.10792/) (2022)
- [Twin Contrastive Learning for Online Clustering](https://mlsanity.com/paper/2210.11680/) (2022)
- [MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features](https://mlsanity.com/paper/2307.12698/) (2023)
- [PASS: An ImageNet replacement for self-supervised pretraining without humans](https://mlsanity.com/paper/2109.13228/) (2021)
- [Soft Contrastive Learning for Time Series](https://mlsanity.com/paper/2312.16424/) (2023)
- [Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach](https://mlsanity.com/paper/2405.15613/) (2024)
- [Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing](https://mlsanity.com/paper/2110.15128/) (2021)

## Abstract (arXiv)

Self-supervised learning algorithms based on instance discrimination train encoders to be invariant to pre-defined transformations of the same instance. While most methods treat different views of the same image as positives for a contrastive loss, we are interested in using positives from other instances in the dataset. Our method, Nearest-Neighbor Contrastive Learning of visual Representations (NNCLR), samples the nearest neighbors from the dataset in the latent space, and treats them as positives. This provides more semantic variations than pre-defined transformations. We find that using the nearest-neighbor as positive in contrastive losses improves performance significantly on ImageNet classification, from 71.7% to 75.6%, outperforming previous state-of-the-art methods. On semi-supervised learning benchmarks we improve performance significantly when only 1% ImageNet labels are available, from 53.8% to 56.5%. On transfer learning benchmarks our method outperforms state-of-the-art methods (including supervised learning with ImageNet) on 8 out of 12 downstream datasets. Furthermore, we demonstrate empirically that our method is less reliant on complex data augmentations. We see a relative reduction of only 2.1% ImageNet Top-1 accuracy when we train using only random crops.