mlsanityData to 7 Oct 2026

Identifiability Results for Multimodal Contrastive Learning

arXiv 2303.09166 ↗First posted 16 Mar 2023cs.LG stat.MLcited by 58

Not on the map. Why?

Abstract (arXiv)

Contrastive learning is a cornerstone underlying recent progress in multi-view and multimodal learning, e.g., in representation learning with image/caption pairs. While its effectiveness is not yet fully understood, a line of recent work reveals that contrastive learning can invert the data generating process and recover ground truth latent factors shared between views. In this work, we present new identifiability results for multimodal contrastive learning, showing that it is possible to recover shared factors in a more general setup than the multi-view setting studied previously. Specifically, we distinguish between the multi-view setting with one generative mechanism (e.g., multiple cameras of the same type) and the multimodal setting that is characterized by distinct mechanisms (e.g., cameras and microphones). Our work generalizes previous identifiability results by redefining the generative process in terms of distinct mechanisms with modality-specific latent variables. We prove that contrastive learning can block-identify latent factors shared between modalities, even when there are nontrivial dependencies between factors. We empirically verify our identifiability results with numerical simulations and corroborate our findings on a complex multimodal dataset of image/text pairs. Zooming out, our work provides a theoretical basis for multimodal representation learning and explains in which settings multimodal contrastive learning can be effective in practice.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 48 of its 68 references that are in the map. First posted on arXiv 16 Mar 2023.

Builds on

48 of 68 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  10. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  11. 2019Contrastive Multiview Codingcited by 1,297
  12. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
36 more
  1. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  2. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  3. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  4. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  5. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  6. 2019On Variational Bounds of Mutual Informationcited by 550
  7. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 494
  8. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  9. 2018Towards a Definition of Disentangled Representationscited by 294
  10. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  11. 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 278
  12. 2016Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAcited by 270
  13. 2023Multimodal Deep Learningcited by 257
  14. 2020Weakly-Supervised Disentanglement Without Compromisescited by 237
  15. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  16. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  17. 2020Representation Learning via Invariant Causal Mechanismscited by 197
  18. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
  19. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
  20. 2021Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICAcited by 158
  21. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  22. 2022Weakly supervised causal representation learningcited by 140
  23. 2022CITRIS: Causal Identifiability from Temporal Intervened Sequencescited by 132
  24. 2020Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Codingcited by 121
  25. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 107
  26. 2016Joint Multimodal Learning with Deep Generative Modelscited by 87
  27. 2019The Incomplete Rosetta Stone Problem: Identifiability Results for Multi-View Nonlinear ICAcited by 86
  28. 2019Weakly Supervised Disentanglement with Guaranteescited by 82
  29. 2022Weakly Supervised Representation Learning with Sparse Perturbationscited by 64
  30. 2016Deep Variational Canonical Correlation Analysiscited by 44
  31. 2022Geometric Multimodal Contrastive Representation Learningcited by 40
  32. 2021Understanding Latent Correlation-Based Multiview Learning and Self-Supervision: An Identifiability Perspectivecited by 37
  33. 2021An Information-theoretic Approach to Distribution Shiftscited by 25
  34. 2019Weakly Supervised Disentanglement by Pairwise Similaritiescited by 22
  35. 2013Nonparametric Estimation of Multi-View Latent Variable Modelscited by 7
  36. 2017Acoustic Feature Learning via Deep Variational Canonical Correlation Analysiscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 15; 2025: 17; 2026: 17.

58 papers.

Sort:

By area:

  1. 2023Multi-View Causal Representation Learning with Partial Observabilitycited by 72
  2. 2023Causal Component Analysiscited by 56
  3. 2024A Sober Look at the Robustness of CLIPs to Spurious Featurescited by 38
  4. 2023Beyond Normal: On the Evaluation of Mutual Information Estimatorscited by 37
  5. 2023Causal Representation Learning Made Identifiable by Grouping of Observational Variablescited by 30
  6. 2024Unifying Causal Representation Learning with the Invariance Principlecited by 30
  7. 2024Marrying Causal Representation Learning with Dynamical Systems for Sciencecited by 22
  8. 2023Multimodal Foundation Models for Material Property Prediction and Discoverycited by 19
  9. 2023Self-Supervised Disentanglement by Leveraging Structure in Data Augmentationscited by 16
  10. 2025SmartCLIP: Modular Vision-language Alignment with Identification Guaranteescited by 14
  11. 2023Event-Based Contrastive Learning for Medical Time Seriescited by 13
  12. 2024Identifiable Causal Representation Learning: Unsupervised, Multi-View, and Multi-Environmentcited by 12
38 more
  1. 2024Causal Representation Learning from Multimodal Biomedical Observationscited by 12
  2. 2023CLAP: Isolating Content from Style through Contrastive Learning with Augmented Promptscited by 10
  3. 2024On the Comparison between Multi-modal and Single-modal Contrastive Learningcited by 10
  4. 2024Beyond DAGs: A Latent Partial Causal Model for Multimodal Learningcited by 9
  5. 2025On the Value of Cross-Modal Misalignment in Multimodal Representation Learningcited by 9
  6. 2025Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Researchcited by 9
  7. 2024Causal Contrastive Learning for Counterfactual Regression Over Timecited by 7
  8. 2025What is causal about causal models and representations?cited by 6
  9. 2024Towards Identifiable Unsupervised Domain Translation: A Diversified Distribution Matching Approachcited by 5
  10. 2024A Probabilistic Model Behind Self-Supervised Learningcited by 5
  11. 2025Sanity Checking Causal Representation Learning on a Simple Real-World Systemcited by 5
  12. 2023Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectivescited by 4
  13. 2024From Orthogonality to Dependency: Learning Disentangled Representation for Multi-Modal Time-Series Sensing Signalscited by 4
  14. 2025A Statistical Theory of Contrastive Learning via Approximate Sufficient Statisticscited by 4
  15. 2026Understanding Self-Supervised Learning via Latent Distribution Matchingcited by 4
  16. 2024Learning Invariant Causal Mechanism from Vision-Language Modelscited by 3
  17. 2025Disentangling Long-Short Term State Under Unknown Interventions for Online Time Series Forecastingcited by 3
  18. 2025Robult: Leveraging Redundancy and Modality Specific Features for Robust Multimodal Learningcited by 3
  19. 2025Mechanistic Independence: A Principle for Identifiable Disentangled Representationscited by 3
  20. 2026The Universal Normal Embeddingcited by 3
  21. 2024Content-Style Learning from Unaligned Domains: Identifiability under Unknown Latent Dimensionscited by 2
  22. 2025Online Time Series Forecasting with Theoretical Guaranteescited by 2
  23. 2025Understanding Hardness of Vision-Language Compositionality from A Token-level Causal Lenscited by 2
  24. 2026The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergencecited by 2
  25. 2024A Unified Causal View of Instruction Tuningcited by 1
  26. 2025Causal View of Time Series Imputation: Some Identification Results on Missing Mechanismcited by 1
  27. 2025Provable Affine Identifiability of Nonlinear CCA under Latent Distributional Priorscited by 1
  28. 2026When Do Diffusion Models learn to Generate Multiple Objects?cited by 1
  29. 2026Content-Style Identification via Differential Independencecited by 1
  30. 2024Two Is Better Than One: Aligned Representation Pairs for Anomaly Detectioncited by 0
  31. 2025Unsupervised Structural-Counterfactual Generation under Domain Shiftcited by 0
  32. 2025Identifiable Object Representations under Spatial Ambiguitiescited by 0
  33. 2025PersonaX: Multimodal Datasets with LLM-Inferred Behavior Traitscited by 0
  34. 2025Eigenfunction Extraction for Ordered Representation Learningcited by 0
  35. 2026Provable Subspace Identification of Nonlinear Multi-view CCAcited by 0
  36. 2026Identifiable Multimodal Causal Representation Learning under Partial Latent Sharingcited by 0
  37. 2026Discrete Causal Representations from Heterogeneous Domains: A Bayesian Approach with Social Survey Applicationscited by 0
  38. 2026Next-Token Prediction Learns Generalisable Representations of Sleep Physiologycited by 0

50 most cited of 58 All 58 (JSON)