# Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embedding

arXiv: https://arxiv.org/abs/2205.14814 · HTML: https://mlsanity.com/paper/2205.14814/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 82 references that are in the map.
- First posted on arXiv 30 May 2022.

## Builds on (most cited first, 30 of 60)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [Contrastive Clustering](https://mlsanity.com/paper/2009.09687/) (2020)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)

## Built on by (most cited first, 28 of 28)

- [Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training](https://mlsanity.com/paper/2305.13689/) (2023)
- [From t-SNE to UMAP with contrastive learning](https://mlsanity.com/paper/2206.01816/) (2022)
- [Contrastive Learning Is Spectral Clustering On Similarity Graph](https://mlsanity.com/paper/2303.15103/) (2023)
- [Matrix Information Theory for Self-Supervised Learning](https://mlsanity.com/paper/2305.17326/) (2023)
- [Complexity Matters: Rethinking the Latent Space for Generative Modeling](https://mlsanity.com/paper/2307.08283/) (2023)
- [Rethinking Weak Supervision in Helping Contrastive Learning](https://mlsanity.com/paper/2306.04160/) (2023)
- [Unsupervised visualization of image datasets using contrastive learning](https://mlsanity.com/paper/2210.09879/) (2022)
- [Information Flow in Self-Supervised Learning](https://mlsanity.com/paper/2309.17281/) (2023)
- [I-Con: A Unifying Framework for Representation Learning](https://mlsanity.com/paper/2504.16929/) (2025)
- [Explore and Exploit the Diverse Knowledge in Model Zoo for Domain Generalization](https://mlsanity.com/paper/2306.02595/) (2023)
- [ArCL: Enhancing Contrastive Learning with Augmentation-Robust Representations](https://mlsanity.com/paper/2303.01092/) (2023)
- [Augmentation Component Analysis: Modeling Similarity via the Augmentation Overlaps](https://mlsanity.com/paper/2206.00471/) (2022)
- [Deciphering the Projection Head: Representation Evaluation Self-supervised Learning](https://mlsanity.com/paper/2301.12189/) (2023)
- [The Shape of Attraction in UMAP: Exploring the Embedding Forces in Dimensionality Reduction](https://mlsanity.com/paper/2503.09101/) (2025)
- [An Augmentation Overlap Theory of Contrastive Learning](https://mlsanity.com/paper/2511.03114/) (2025)
- [Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion](https://mlsanity.com/paper/2402.16305/) (2024)
- [Navigating the Effect of Parametrization for Dimensionality Reduction](https://mlsanity.com/paper/2411.15894/) (2024)
- [An Augmentation-Aware Theory for Self-Supervised Contrastive Learning](https://mlsanity.com/paper/2505.22196/) (2025)
- [Difficult Examples Hurt Unsupervised Contrastive Learning: A Theoretical Perspective](https://mlsanity.com/paper/2501.01317/) (2025)
- [FedNE: Surrogate-Assisted Federated Neighbor Embedding for Dimensionality Reduction](https://mlsanity.com/paper/2409.11509/) (2024)
- [Node Embeddings via Neighbor Embeddings](https://mlsanity.com/paper/2503.23822/) (2025)
- [MAPLE: Self-Supervised Learning-Enhanced Nonlinear Dimensionality Reduction for Visual Analysis](https://mlsanity.com/paper/2601.20173/) (2026)
- [On the Generalization and Causal Explanation in Self-Supervised Learning](https://mlsanity.com/paper/2410.00772/) (2024)
- [Pseudo-Label Enhanced Prototypical Contrastive Learning for Uniformed Intent Discovery](https://mlsanity.com/paper/2410.20219/) (2024)
- [Group Contrastive Learning for Weakly Paired Multimodal Data](https://mlsanity.com/paper/2602.04021/) (2026)
- [UMAP Is Spectral Clustering on the Fuzzy Nearest-Neighbor Graph](https://mlsanity.com/paper/2602.11662/) (2026)
- [Maximizing Incremental Information Entropy for Contrastive Learning](https://mlsanity.com/paper/2603.12594/) (2026)
- [Say, Echo, Do: Strategic Narratives and Revealed Positioning in Financial Markets](https://mlsanity.com/paper/2609.38545/) (2026)

## Abstract (arXiv)

Contrastive learning, especially self-supervised contrastive learning (SSCL), has achieved great success in extracting powerful features from unlabeled data. In this work, we contribute to the theoretical understanding of SSCL and uncover its connection to the classic data visualization method, stochastic neighbor embedding (SNE), whose goal is to preserve pairwise distances. From the perspective of preserving neighboring information, SSCL can be viewed as a special case of SNE with the input space pairwise similarities specified by data augmentation. The established correspondence facilitates deeper theoretical understanding of learned features of SSCL, as well as methodological guidelines for practical improvement. Specifically, through the lens of SNE, we provide novel analysis on domain-agnostic augmentations, implicit bias and robustness of learned features. To illustrate the practical advantage, we demonstrate that the modifications from SNE to t-SNE can also be adopted in the SSCL setting, achieving significant improvement in both in-distribution and out-of-distribution generalization.