mlsanityData to 14 Oct 2026

Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data

arXiv 2401.08567 ↗First posted 16 Jan 2024cs.LG cs.AI cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

Building cross-modal applications is challenging due to limited paired multi-modal data. Recent works have shown that leveraging a pre-trained multi-modal contrastive representation space enables cross-modal tasks to be learned from uni-modal data. This is based on the assumption that contrastive optimization makes embeddings from different modalities interchangeable. However, this assumption is under-explored due to the poorly understood geometry of the multi-modal contrastive space, where a modality gap exists. In our study, we provide a theoretical explanation of this space's geometry and introduce a three-step method, C3 (Connect, Collapse, Corrupt), to bridge the modality gap, enhancing the interchangeability of embeddings. Our C3 method significantly improves cross-modal learning from uni-modal data, achieving state-of-the-art results on zero-shot image / audio / video captioning and text-to-image generation.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

32 of 42 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  7. 2016Improved Techniques for Training GANscited by 2,970
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  9. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  10. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  11. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  12. 2021Florence: A New Foundation Model for Computer Visioncited by 804
20 more
  1. 2021Robust fine-tuning of zero-shot modelscited by 797
  2. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  3. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  4. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  5. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  6. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  7. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  8. 2019Clotho: An Audio Captioning Datasetcited by 358
  9. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  10. 2022Domino: Discovering Systematic Errors with Cross-Modal Embeddingscited by 156
  11. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  12. 2022Natural Language Descriptions of Deep Visual Featurescited by 137
  13. 2022Text-Only Training for Image Captioning using Noise-Injected CLIPcited by 91
  14. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 90
  15. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
  16. 2023Diagnosing and Rectifying Vision Models using Languagecited by 58
  17. 2022Shifted Diffusion for Text-to-image Generationcited by 37
  18. 2023Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generationcited by 35
  19. 2022Multimodal Knowledge Alignment with Reinforcement Learningcited by 30
  20. 2022Lafite2: Few-shot Text-to-Image Generationcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 12; 2026: 11.

27 papers.

Sort:

By area:

  1. 2024Improving Medical Multi-modal Contrastive Learning with Expert Annotationscited by 16
  2. 2025Closing the Modality Gap for Mixed Modality Searchcited by 12
  3. 2024TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignmentcited by 10
  4. 2026Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Modelscited by 10
  5. 2025Praxis-VLM: Vision-Grounded Decision Making via Text-Driven Reinforcement Learningcited by 9
  6. 2025Fine-Grained Captioning of Long Videos through Scene Graph Consolidationcited by 8
  7. 2024Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learningcited by 6
  8. 2026Is the Modality Gap a Bug or a Feature? A Robustness Perspectivecited by 5
  9. 2025Unified modality separation: A vision-language framework for unsupervised domain adaptationcited by 4
  10. 2025Text-Only Data Synthesis for Vision Language Model Trainingcited by 3
  11. 2025Diffusion-Link: Diffusion Probabilistic Model for Bridging the Audio-Text Modality Gapcited by 3
  12. 2025Learning by Neighbor-Aware Semantics, Deciding by Open-form Flows: Towards Robust Zero-Shot Skeleton Action Recognitioncited by 3
15 more
  1. 2025Learning to Match Unpaired Data with Minimum Entropy Couplingcited by 2
  2. 2026TextME: Bridging Unseen Modalities Through Text Descriptionscited by 2
  3. 2026Anisotropic Modality Aligncited by 2
  4. 2025Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Modelscited by 1
  5. 2024ResidualDroppath: Enhancing Feature Reuse over Residual Connectionscited by 0
  6. 2025Beyond the Linear Separability Ceiling: Aligning Representations in VLMscited by 0
  7. 2025SGCap: Decoding Semantic Group for Zero-shot Video Captioningcited by 0
  8. 2025Adapting MLLMs for Nuanced Video Retrievalcited by 0
  9. 2026CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebookcited by 0
  10. 2026COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddingscited by 0
  11. 2026Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoderscited by 0
  12. 2026Entity-Faithful Repair of Synthetic Supervision for Zero-Shot Image Captioningcited by 0
  13. 2026Watching Synthetic Videos: Aligning Cross-modal Representations with Visual Synthesis for Zero-shot Video Captioningcited by 0
  14. 2026Beyond Multimodal Alignment: Certifying Physical Language through Response Substitution and Ordered Executioncited by 0
  15. 2026Learning Through Game: Skewed Transfer of Tabular Knowledge to Strengthen Image Modelcited by 0