mlsanityData to 7 Oct 2026

Contrastive Learning Inverts the Data Generating Process

arXiv 2102.08850 ↗First posted 17 Feb 2021cs.LG cs.CVcited by 220

Not on the map. Why?

Abstract (arXiv)

Contrastive learning has recently seen tremendous success in self-supervised learning. So far, however, it is largely unclear why the learned representations generalize so effectively to a large variety of downstream tasks. We here prove that feedforward models trained with objectives belonging to the commonly used InfoNCE family learn to implicitly invert the underlying generative model of the observed data. While the proofs make certain statistical assumptions about the generative model, we observe empirically that our findings hold even if these assumptions are severely violated. Our theory highlights a fundamental connection between contrastive learning, generative modeling, and nonlinear independent component analysis, thereby furthering our understanding of the learned representations as well as providing a theoretical foundation to derive more effective contrastive losses.

Key facts. Cited by 220 papers in the map (data through 7 Oct 2026). Builds on 33 of its 65 references that are in the map. First posted on arXiv 17 Feb 2021.

Builds on

33 of 65 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2017Billion-scale similarity search with GPUscited by 2,477
  8. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  9. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  10. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  12. 2019Contrastive Multiview Codingcited by 1,297
21 more
  1. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  2. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  3. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  4. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  5. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  6. 2020Contrastive Learning with Hard Negative Samplescited by 619
  7. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  8. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 494
  9. 2020Debiased Contrastive Learningcited by 354
  10. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  11. 2019On Mutual Information Maximization for Representation Learningcited by 315
  12. 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 278
  13. 2016Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAcited by 270
  14. 2020Weakly-Supervised Disentanglement Without Compromisescited by 237
  15. 2018An efficient framework for learning sentence representationscited by 197
  16. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  17. 2020Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Codingcited by 121
  18. 2020On Linear Identifiability of Learned Representationscited by 103
  19. 2020ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICAcited by 102
  20. 2019On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Datasetcited by 100
  21. 2020On the Transfer of Disentangled Representations in Realistic Settingscited by 63

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 40; 2023: 51; 2024: 41; 2025: 45; 2026: 30.

220 papers.

Sort:

By area:

  1. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  2. 2024The Platonic Representation Hypothesiscited by 362
  3. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  4. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  5. 2021Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICAcited by 158
  6. 2022Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphscited by 144
  7. 2022Interventional Causal Representation Learningcited by 140
  8. 2022CITRIS: Causal Identifiability from Temporal Intervened Sequencescited by 132
  9. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  10. 2022Learnable latent embeddings for joint behavioral and neural analysiscited by 113
  11. 2023Nonparametric Identifiability of Causal Representations from Unknown Interventionscited by 105
  12. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 100
38 more
  1. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  2. 2023Learning Linear Causal Representations from Interventions under General Nonlinear Mixingcited by 92
  3. 2023Collaborative Perception in Autonomous Driving: Methods, Datasets and Challengescited by 87
  4. 2022Linear Causal Disentanglement via Interventionscited by 84
  5. 2023Identifiability Guarantees for Causal Disentanglement from Soft Interventionscited by 84
  6. 2021Self-Supervised Learning with Kernel Dependence Maximizationcited by 77
  7. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  8. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  9. 2023Multi-View Causal Representation Learning with Partial Observabilitycited by 72
  10. 2021Unsupervised Learning of Compositional Energy Conceptscited by 71
  11. 2022Identifiability of deep generative models without auxiliary informationcited by 70
  12. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 67
  13. 2022Weakly Supervised Representation Learning with Sparse Perturbationscited by 64
  14. 2023Compositional Generalization from First Principlescited by 62
  15. 2023Concept Algebra for (Score-Based) Text-Controlled Generative Modelscited by 59
  16. 2023Identifiability of latent-variable and structural-equation models: from linear to nonlinearcited by 58
  17. 2023Identifiability Results for Multimodal Contrastive Learningcited by 58
  18. 2022Causal Representation Learning for Instantaneous and Temporal Effects in Interactive Systemscited by 50
  19. 2023Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learningcited by 48
  20. 2021Learning Disentangled Representations in the Imaging Domaincited by 46
  21. 2025The Origins of Representation Manifolds in Large Language Modelscited by 46
  22. 2023Provably Learning Object-Centric Representationscited by 45
  23. 2022Neural Manifold Clustering and Embeddingcited by 44
  24. 2021A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectivescited by 41
  25. 2023Self-supervised learning of Split Invariant Equivariant representationscited by 41
  26. 2022Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learningcited by 40
  27. 2022Do More Negative Samples Necessarily Hurt in Contrastive Learning?cited by 39
  28. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 39
  29. 2021Understanding Latent Correlation-Based Multiview Learning and Self-Supervision: An Identifiability Perspectivecited by 37
  30. 2022On the Power of Foundation Modelscited by 37
  31. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  32. 2023Beyond Normal: On the Evaluation of Mutual Information Estimatorscited by 37
  33. 2023Learning nonparametric latent causal graphs with unknown interventionscited by 34
  34. 2025Shared Global and Local Geometry of Language Model Embeddingscited by 34
  35. 2022Non-isotropy Regularization for Proxy-based Deep Metric Learningcited by 33
  36. 2024Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuningcited by 31
  37. 2021An Image is Worth More Than a Thousand Words: Towards Disentanglement in the Wildcited by 30
  38. 2021Properties from Mechanisms: An Equivariance Perspective on Identifiable Representation Learningcited by 30

50 most cited of 220 All 220 (JSON)