# Is the Modality Gap a Bug or a Feature? A Robustness Perspective

arXiv: https://arxiv.org/abs/2603.29080 · HTML: https://mlsanity.com/paper/2603.29080/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 44 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [Scaling up GANs for Text-to-Image Synthesis](https://mlsanity.com/paper/2303.05511/) (2023)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Self-regulating Prompts: Foundational Model Adaptation without Forgetting](https://mlsanity.com/paper/2307.06948/) (2023)
- [Demystifying CLIP Data](https://mlsanity.com/paper/2309.16671/) (2023)
- [On the Adversarial Robustness of Multi-Modal Foundation Models](https://mlsanity.com/paper/2308.10741/) (2023)
- [Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)](https://mlsanity.com/paper/2205.01397/) (2022)
- [Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models](https://mlsanity.com/paper/2402.12336/) (2024)
- [CLIPood: Generalizing CLIP to Out-of-Distributions](https://mlsanity.com/paper/2302.00864/) (2023)
- [Diagnosing and Rectifying Vision Models using Language](https://mlsanity.com/paper/2302.04269/) (2023)
- [Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models](https://mlsanity.com/paper/2404.07983/) (2024)
- [Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning](https://mlsanity.com/paper/2303.05952/) (2023)
- [A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)](https://mlsanity.com/paper/2402.07410/) (2024)
- [Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion](https://mlsanity.com/paper/2502.04263/) (2025)
- [Geodesic Multi-Modal Mixup for Robust Fine-Tuning](https://mlsanity.com/paper/2203.03897/) (2022)
- [Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data](https://mlsanity.com/paper/2401.08567/) (2024)
- [Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learning](https://mlsanity.com/paper/2412.07909/) (2024)
- [ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations](https://mlsanity.com/paper/2312.04655/) (2023)
- [Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts](https://mlsanity.com/paper/2310.20159/) (2023)
- [Mutual-modality Adversarial Attack with Semantic Perturbation](https://mlsanity.com/paper/2312.12768/) (2023)

## Built on by (most cited first, 5 of 5)

- [Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers](https://mlsanity.com/paper/2605.14893/) (2026)
- [Learning to See Locally and Align Clinically with Pathology Semantics for Radiology Report Generation](https://mlsanity.com/paper/2608.00279/) (2026)
- [LAYERSCOPE: A Layerwise Characterization of Video and Multimodal Learned Representations](https://mlsanity.com/paper/2609.28086/) (2026)
- [One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models](https://mlsanity.com/paper/2609.36101/) (2026)
- [Omni-Modal LLMs Know More Than They Say](https://mlsanity.com/paper/2610.16135/) (2026)

## Abstract (arXiv)

Many modern multi-modal models (e.g. CLIP) seek an embedding space in which the two modalities are aligned. Somewhat surprisingly, almost all existing models show a strong modality gap: the distribution of images is well-separated from the distribution of texts in the shared embedding space. Despite a series of recent papers on this topic, it is still not clear why this gap exists nor whether closing the gap in post-processing will lead to better performance on downstream tasks. In this paper we show that under certain conditions, minimizing the contrastive loss yields a representation in which the two modalities are separated by a global gap vector that is orthogonal to their embeddings. We also show that under these conditions the modality gap is monotonically related to robustness: decreasing the gap does not change the clean accuracy of the models but makes it less likely that a model will change its output when the embeddings are perturbed. Our experiments show that for many real-world VLMs we can significantly increase robustness by a simple post-processing step that moves one modality towards the mean of the other modality, without any loss of clean accuracy.