# Identifiability Results for Multimodal Contrastive Learning

arXiv: https://arxiv.org/abs/2303.09166 · HTML: https://mlsanity.com/paper/2303.09166/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 68 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style](https://mlsanity.com/paper/2106.04619/) (2021)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning](https://mlsanity.com/paper/1805.08651/) (2018)
- [Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA](https://mlsanity.com/paper/1605.06336/) (2016)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [Weakly-Supervised Disentanglement Without Compromises](https://mlsanity.com/paper/2002.02886/) (2020)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Representation Learning via Invariant Causal Mechanisms](https://mlsanity.com/paper/2010.07922/) (2020)
- [Learning Robust Representations via Multi-View Information Bottleneck](https://mlsanity.com/paper/2002.07017/) (2020)

## Built on by (most cited first, 30 of 61)

- [Multi-View Causal Representation Learning with Partial Observability](https://mlsanity.com/paper/2311.04056/) (2023)
- [Causal Component Analysis](https://mlsanity.com/paper/2305.17225/) (2023)
- [A Sober Look at the Robustness of CLIPs to Spurious Features](https://mlsanity.com/paper/2403.11497/) (2024)
- [Beyond Normal: On the Evaluation of Mutual Information Estimators](https://mlsanity.com/paper/2306.11078/) (2023)
- [Causal Representation Learning Made Identifiable by Grouping of Observational Variables](https://mlsanity.com/paper/2310.15709/) (2023)
- [Unifying Causal Representation Learning with the Invariance Principle](https://mlsanity.com/paper/2409.02772/) (2024)
- [Marrying Causal Representation Learning with Dynamical Systems for Science](https://mlsanity.com/paper/2405.13888/) (2024)
- [Multimodal Foundation Models for Material Property Prediction and Discovery](https://mlsanity.com/paper/2312.00111/) (2023)
- [Self-Supervised Disentanglement by Leveraging Structure in Data Augmentations](https://mlsanity.com/paper/2311.08815/) (2023)
- [SmartCLIP: Modular Vision-language Alignment with Identification Guarantees](https://mlsanity.com/paper/2507.22264/) (2025)
- [Event-Based Contrastive Learning for Medical Time Series](https://mlsanity.com/paper/2312.10308/) (2023)
- [Causal Representation Learning from Multimodal Biomedical Observations](https://mlsanity.com/paper/2411.06518/) (2024)
- [Identifiable Causal Representation Learning: Unsupervised, Multi-View, and Multi-Environment](https://mlsanity.com/paper/2406.13371/) (2024)
- [CLAP: Isolating Content from Style through Contrastive Learning with Augmented Prompts](https://mlsanity.com/paper/2311.16445/) (2023)
- [Beyond DAGs: A Latent Partial Causal Model for Multimodal Learning](https://mlsanity.com/paper/2402.06223/) (2024)
- [On the Comparison between Multi-modal and Single-modal Contrastive Learning](https://mlsanity.com/paper/2411.02837/) (2024)
- [On the Value of Cross-Modal Misalignment in Multimodal Representation Learning](https://mlsanity.com/paper/2504.10143/) (2025)
- [Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research](https://mlsanity.com/paper/2504.13101/) (2025)
- [Causal Contrastive Learning for Counterfactual Regression Over Time](https://mlsanity.com/paper/2406.00535/) (2024)
- [What is causal about causal models and representations?](https://mlsanity.com/paper/2501.19335/) (2025)
- [Towards Identifiable Unsupervised Domain Translation: A Diversified Distribution Matching Approach](https://mlsanity.com/paper/2401.09671/) (2024)
- [A Probabilistic Model Behind Self-Supervised Learning](https://mlsanity.com/paper/2402.01399/) (2024)
- [Sanity Checking Causal Representation Learning on a Simple Real-World System](https://mlsanity.com/paper/2502.20099/) (2025)
- [A Statistical Theory of Contrastive Learning via Approximate Sufficient Statistics](https://mlsanity.com/paper/2503.17538/) (2025)
- [Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives](https://mlsanity.com/paper/2309.00380/) (2023)
- [From Orthogonality to Dependency: Learning Disentangled Representation for Multi-Modal Time-Series Sensing Signals](https://mlsanity.com/paper/2405.16083/) (2024)
- [Understanding Self-Supervised Learning via Latent Distribution Matching](https://mlsanity.com/paper/2605.03517/) (2026)
- [Learning Invariant Causal Mechanism from Vision-Language Models](https://mlsanity.com/paper/2405.15289/) (2024)
- [Disentangling Long-Short Term State Under Unknown Interventions for Online Time Series Forecasting](https://mlsanity.com/paper/2502.12603/) (2025)
- [Robult: Leveraging Redundancy and Modality Specific Features for Robust Multimodal Learning](https://mlsanity.com/paper/2509.03477/) (2025)

## Abstract (arXiv)

Contrastive learning is a cornerstone underlying recent progress in multi-view and multimodal learning, e.g., in representation learning with image/caption pairs. While its effectiveness is not yet fully understood, a line of recent work reveals that contrastive learning can invert the data generating process and recover ground truth latent factors shared between views. In this work, we present new identifiability results for multimodal contrastive learning, showing that it is possible to recover shared factors in a more general setup than the multi-view setting studied previously. Specifically, we distinguish between the multi-view setting with one generative mechanism (e.g., multiple cameras of the same type) and the multimodal setting that is characterized by distinct mechanisms (e.g., cameras and microphones). Our work generalizes previous identifiability results by redefining the generative process in terms of distinct mechanisms with modality-specific latent variables. We prove that contrastive learning can block-identify latent factors shared between modalities, even when there are nontrivial dependencies between factors. We empirically verify our identifiability results with numerical simulations and corroborate our findings on a complex multimodal dataset of image/text pairs. Zooming out, our work provides a theoretical basis for multimodal representation learning and explains in which settings multimodal contrastive learning can be effective in practice.