mlsanityData to 7 Oct 2026

InfoNCE: Identifying the Gap Between Theory and Practice

arXiv 2407.00143 ↗First posted 28 Jun 2024cs.LG cs.CV stat.MLcited by 39

Not on the map. Why?

Abstract (arXiv)

Prior theory work on Contrastive Learning via the InfoNCE loss showed that, under certain assumptions, the learned representations recover the ground-truth latent factors. We argue that these theories overlook crucial aspects of how CL is deployed in practice. Specifically, they either assume equal variance across all latents or that certain latents are kept invariant. However, in practice, positive pairs are often generated using augmentations such as strong cropping to just a few pixels. Hence, a more realistic assumption is that all latent factors change with a continuum of variability across all factors. We introduce AnInfoNCE, a generalization of InfoNCE that can provably uncover the latent factors in this anisotropic setting, broadly generalizing previous identifiability results in CL. We validate our identifiability results in controlled experiments and show that AnInfoNCE increases the recovery of previously collapsed information in CIFAR10 and ImageNet, albeit at the cost of downstream accuracy. Finally, we discuss the remaining mismatches between theoretical assumptions and practical implementations.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 57 of its 93 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

57 of 93 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  8. 2020Supervised Contrastive Learningcited by 2,803
  9. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  12. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
38 more
  1. 2020Contrastive Learning with Hard Negative Samplescited by 619
  2. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  3. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  4. 2020Understanding the Behaviour of Contrastive Losscited by 397
  5. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  6. 2020Debiased Contrastive Learningcited by 354
  7. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  8. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  9. 2016Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAcited by 270
  10. 2020What Should Not Be Contrastive in Contrastive Learningcited by 224
  11. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  12. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  13. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  14. 2020Intriguing Properties of Contrastive Lossescited by 129
  15. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  16. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  17. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  18. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  19. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
  20. 2020On Linear Identifiability of Learned Representationscited by 103
  21. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  22. 2019The Incomplete Rosetta Stone Problem: Identifiability Results for Multi-View Nonlinear ICAcited by 86
  23. 2021Equivariant Contrastive Learningcited by 65
  24. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 57
  25. 2020On Mutual Information in Contrastive Learning for Visual Representationscited by 54
  26. 2021Improving Transferability of Representations via Augmentation-Aware Self-Supervisioncited by 48
  27. 2022Robust Contrastive Learning against Noisy Viewscited by 45
  28. 2022Rethinking the Augmentation Module in Contrastive Learning: Learning Hierarchical Augmentation Invariance with Expanded Viewscited by 37
  29. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  30. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
  31. 2023Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputscited by 28
  32. 2023Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regressioncited by 20
  33. 2022Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learningcited by 19
  34. 2022A Non-isotropic Probabilistic Take on Proxy-based Deep Metric Learningcited by 19
  35. 2019Optimal experimental design via Bayesian optimization: active causal structure learning for Gaussian process networkscited by 18
  36. 2022Toward a Geometrical Understanding of Self-supervised Contrastive Learningcited by 17
  37. 2022On the Importance of Hyperparameters and Data Augmentation for Self-Supervised Learningcited by 16
  38. 2023Self-Supervised Disentanglement by Leveraging Structure in Data Augmentationscited by 16

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 17; 2026: 19.

39 papers.

Sort:

By area:

  1. 2025DIVER: A Multi-Stage Approach for Reasoning-intensive Information Retrievalcited by 35
  2. 2024Cross-Entropy Is All You Need To Invert the Data Generating Processcited by 30
  3. 2025RzenEmbed: Towards Comprehensive Multimodal Retrievalcited by 28
  4. 2026When Does LeJEPA Learn a World Model?cited by 27
  5. 2025From superposition to sparse codes: interpretable representations in neural networkscited by 16
  6. 2024Self-supervised contrastive learning performs non-linear system identificationcited by 9
  7. 2025Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Researchcited by 9
  8. 2026OneReason Technical Reportcited by 8
  9. 2025seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Modelscited by 7
  10. 2026X-Tokenizer: A Multimodal Action Tokenizer for Vision-Language-Action Pretrainingcited by 6
  11. 2025Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learningcited by 4
  12. 2025Self-Supervised Frameworks for Speaker Verification via Bootstrapped Positive Samplingcited by 3
27 more
  1. 2025MSPL: Multi-Step Pseudo-Labeling for Open-Vocabulary Object Detectioncited by 2
  2. 2025Structure is Supervision: Multiview Masked Autoencoders for Radiologycited by 2
  3. 2024SimO Loss: Anchor-Free Contrastive Loss for Fine-Grained Supervised Contrastive Learningcited by 1
  4. 2025Combining Textual and Structural Information for Premise Selection in Leancited by 1
  5. 2025Learning Fair Graph Representations with Multi-view Information Bottleneckcited by 1
  6. 2026Self-Supervised Learning from Structural Invariancecited by 1
  7. 2026Learning Accurate Segmentation Purely from Self-Supervisioncited by 1
  8. 2026Asymptotic and Finite-Time Guarantees for Langevin-Based Temperature Annealing in InfoNCEcited by 1
  9. 2026A Multihead Continual Learning Framework for Fine-Grained Fashion Image Retrieval with Contrastive Learning and Exponential Moving Average Distillationcited by 1
  10. 2025AMPED: Adaptive Multi-objective Projection for balancing Exploration and skill Diversificationcited by 0
  11. 2025Zero Shot Composed Image Retrievalcited by 0
  12. 2025Contrastive Self-Supervised Network Intrusion Detection using Augmented Negative Pairscited by 0
  13. 2025Empowering Denoising Sequential Recommendation with Large Language Model Embeddingscited by 0
  14. 2025Eigenfunction Extraction for Ordered Representation Learningcited by 0
  15. 2025SoilX: Calibration-Free Comprehensive Soil Sensing through Contrastive Cross-Component Learningcited by 0
  16. 2026Disentanglement Beyond Generative Models with Riemannian ICAcited by 0
  17. 2026Dual-Pathway Geometry-Aware MLLM for Spatial Intelligencecited by 0
  18. 2026OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignmentcited by 0
  19. 2026Cross-modal linkage risk in clinical vision-language modelscited by 0
  20. 2026The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learningcited by 0
  21. 2026Expressivity In Multimodal Contrastive Learningcited by 0
  22. 2026Measuring Similarity between Artistic and AI Generated Images using Siamese Neural Networkscited by 0
  23. 2026V2TATC: Joint Voice-Trajectory Embedding and Dataset for Air Traffic Controller Situational Awarenesscited by 0
  24. 2026When Does Text Inform? Benchmarking Information-Theoretic Metrics for Multimodal Time-Series Forecastingcited by 0
  25. 2026Enhancing Shrimp Disease Detection via Deep Learning and Data Refinement for Resilient Aquaculturecited by 0
  26. 2026GraphWrit3R: End-to-End 3D Scene Graph Writingcited by 0
  27. 2026In-Distribution Imagination for Model-Based Offline Reinforcement Learningcited by 0