mlsanityData to 7 Oct 2026

Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning

arXiv 2203.02053 ↗First posted 3 Mar 2022cs.CL cs.AI cs.CVcited by 553

Not on the map. Why?

Cited by 155 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present modality gap, an intriguing geometric phenomenon of the representation space of multi-modal models. Specifically, we show that different data modalities (e.g. images and text) are embedded at arm's length in their shared representation in multi-modal models such as CLIP. Our systematic analysis demonstrates that this gap is caused by a combination of model initialization and contrastive learning optimization. In model initialization, we show empirically and theoretically that the representation of a common deep neural network is restricted to a narrow cone. As a consequence, in a multi-modal model with two encoders, the representations of the two modalities are clearly apart when the model is initialized. During optimization, contrastive learning keeps the different modalities separate by a certain distance, which is influenced by the temperature parameter in the loss function. Our experiments further demonstrate that varying the modality gap distance has a significant impact in improving the model's downstream zero-shot classification performance and fairness. Our code and data are available at https://modalitygap.readthedocs.io/

Key facts. Cited by 553 papers in the map (data through 7 Oct 2026). Builds on 44 of its 58 references that are in the map. First posted on arXiv 3 Mar 2022.

Builds on

44 of 58 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2016Layer Normalizationcited by 5,579
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  12. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
32 more
  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  2. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  6. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  7. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2020Graph Contrastive Learning with Augmentationscited by 1,203
  10. 2017A Closer Look at Memorization in Deep Networkscited by 936
  11. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  12. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  13. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  14. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  15. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  16. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  17. 2017Word Translation Without Parallel Datacited by 480
  18. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
  19. 2020GCC: Graph Contrastive Coding for Graph Neural Network Pre-Trainingcited by 461
  20. 2020Understanding the Behaviour of Contrastive Losscited by 397
  21. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  22. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  23. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  24. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  25. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  26. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  27. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  28. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 140
  29. 2020Parametric UMAP embeddings for representation and semi-supervised learningcited by 103
  30. 2016Learning Visual N-Grams from Web Datacited by 95
  31. 2020On the Sentence Embeddings from Pre-trained Language Modelscited by 83
  32. 2022Delaunay Component Analysis for Evaluation of Data Representationscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 25; 2023: 77; 2024: 143; 2025: 155; 2026: 153.

553 papers.

Sort:

By area:

  1. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  2. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  3. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  4. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
  5. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  6. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
  7. 2023Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priorscited by 156
  8. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  9. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  10. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  11. 2024Exploring scalable medical image encoders beyond text supervisioncited by 129
  12. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 123
38 more
  1. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  2. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 117
  3. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  4. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 116
  5. 2024MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Datacited by 105
  6. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  7. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  8. 2022Text-Only Training for Image Captioning using Noise-Injected CLIPcited by 91
  9. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
  10. 2023Learning without Forgetting for Vision-Language Modelscited by 79
  11. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  12. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 74
  13. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  14. 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 72
  15. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  16. 2024VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidancecited by 65
  17. 2023PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalizationcited by 59
  18. 2023Diagnosing and Rectifying Vision Models using Languagecited by 58
  19. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  20. 2022Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representationscited by 57
  21. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 55
  22. 2025Harnessing the Universal Geometry of Embeddingscited by 54
  23. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 53
  24. 2023Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learningcited by 53
  25. 2023Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Featurescited by 52
  26. 2024Gramian Multimodal Representation Learning and Alignmentcited by 52
  27. 2024Measuring and Controlling Instruction (In)Stability in Language Model Dialogscited by 51
  28. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyondcited by 50
  29. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Modelscited by 48
  30. 2023SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalizationcited by 48
  31. 2022CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Languagecited by 46
  32. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  33. 2024Spherical Linear Interpolation and Text-Anchoring for Zero-shot Composed Image Retrievalcited by 44
  34. 2023Separate Anything You Describecited by 43
  35. 2024The Double-Ellipsoid Geometry of CLIPcited by 42
  36. 2025Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversioncited by 42
  37. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 42
  38. 2023Black Box Few-Shot Adaptation for Vision-Language modelscited by 39

50 most cited of 553 All 553 (JSON)