# Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data

arXiv: https://arxiv.org/abs/2401.08567 · HTML: https://mlsanity.com/paper/2401.08567/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 16 Jan 2024.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [Domino: Discovering Systematic Errors with Cross-Modal Embeddings](https://mlsanity.com/paper/2203.14960/) (2022)
- [ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic](https://mlsanity.com/paper/2111.14447/) (2021)
- [Natural Language Descriptions of Deep Visual Features](https://mlsanity.com/paper/2201.11114/) (2022)
- [Text-Only Training for Image Captioning using Noise-Injected CLIP](https://mlsanity.com/paper/2211.00575/) (2022)
- [Language Models Can See: Plugging Visual Controls in Text Generation](https://mlsanity.com/paper/2205.02655/) (2022)
- [DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training](https://mlsanity.com/paper/2303.03032/) (2023)
- [Diagnosing and Rectifying Vision Models using Language](https://mlsanity.com/paper/2302.04269/) (2023)
- [Shifted Diffusion for Text-to-image Generation](https://mlsanity.com/paper/2211.15388/) (2022)
- [Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generation](https://mlsanity.com/paper/2302.07865/) (2023)

## Built on by (most cited first, 27 of 27)

- [Improving Medical Multi-modal Contrastive Learning with Expert Annotations](https://mlsanity.com/paper/2403.10153/) (2024)
- [Closing the Modality Gap for Mixed Modality Search](https://mlsanity.com/paper/2507.19054/) (2025)
- [TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment](https://mlsanity.com/paper/2405.13911/) (2024)
- [Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models](https://mlsanity.com/paper/2602.07026/) (2026)
- [Praxis-VLM: Vision-Grounded Decision Making via Text-Driven Reinforcement Learning](https://mlsanity.com/paper/2503.16965/) (2025)
- [Fine-Grained Captioning of Long Videos through Scene Graph Consolidation](https://mlsanity.com/paper/2502.16427/) (2025)
- [Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learning](https://mlsanity.com/paper/2410.01529/) (2024)
- [Is the Modality Gap a Bug or a Feature? A Robustness Perspective](https://mlsanity.com/paper/2603.29080/) (2026)
- [Unified modality separation: A vision-language framework for unsupervised domain adaptation](https://mlsanity.com/paper/2508.04987/) (2025)
- [Text-Only Data Synthesis for Vision Language Model Training](https://mlsanity.com/paper/2503.22655/) (2025)
- [Diffusion-Link: Diffusion Probabilistic Model for Bridging the Audio-Text Modality Gap](https://mlsanity.com/paper/2510.11330/) (2025)
- [Learning by Neighbor-Aware Semantics, Deciding by Open-form Flows: Towards Robust Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2511.09388/) (2025)
- [Learning to Match Unpaired Data with Minimum Entropy Coupling](https://mlsanity.com/paper/2503.08501/) (2025)
- [TextME: Bridging Unseen Modalities Through Text Descriptions](https://mlsanity.com/paper/2602.03098/) (2026)
- [Anisotropic Modality Align](https://mlsanity.com/paper/2605.07825/) (2026)
- [Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models](https://mlsanity.com/paper/2512.03463/) (2025)
- [ResidualDroppath: Enhancing Feature Reuse over Residual Connections](https://mlsanity.com/paper/2411.09475/) (2024)
- [Beyond the Linear Separability Ceiling: Aligning Representations in VLMs](https://mlsanity.com/paper/2507.07574/) (2025)
- [SGCap: Decoding Semantic Group for Zero-shot Video Captioning](https://mlsanity.com/paper/2508.01270/) (2025)
- [Adapting MLLMs for Nuanced Video Retrieval](https://mlsanity.com/paper/2512.13511/) (2025)
- [CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebook](https://mlsanity.com/paper/2605.18257/) (2026)
- [COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddings](https://mlsanity.com/paper/2605.29628/) (2026)
- [Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders](https://mlsanity.com/paper/2606.29888/) (2026)
- [Entity-Faithful Repair of Synthetic Supervision for Zero-Shot Image Captioning](https://mlsanity.com/paper/2608.00994/) (2026)
- [Watching Synthetic Videos: Aligning Cross-modal Representations with Visual Synthesis for Zero-shot Video Captioning](https://mlsanity.com/paper/2608.11013/) (2026)
- [Beyond Multimodal Alignment: Certifying Physical Language through Response Substitution and Ordered Execution](https://mlsanity.com/paper/2608.19492/) (2026)
- [Learning Through Game: Skewed Transfer of Tabular Knowledge to Strengthen Image Model](https://mlsanity.com/paper/2609.32272/) (2026)

## Abstract (arXiv)

Building cross-modal applications is challenging due to limited paired multi-modal data. Recent works have shown that leveraging a pre-trained multi-modal contrastive representation space enables cross-modal tasks to be learned from uni-modal data. This is based on the assumption that contrastive optimization makes embeddings from different modalities interchangeable. However, this assumption is under-explored due to the poorly understood geometry of the multi-modal contrastive space, where a modality gap exists. In our study, we provide a theoretical explanation of this space's geometry and introduce a three-step method, C3 (Connect, Collapse, Corrupt), to bridge the modality gap, enhancing the interchangeability of embeddings. Our C3 method significantly improves cross-modal learning from uni-modal data, achieving state-of-the-art results on zero-shot image / audio / video captioning and text-to-image generation.