mlsanityData to 7 Oct 2026

The Double-Ellipsoid Geometry of CLIP

arXiv 2411.14517 ↗First posted 21 Nov 2024cs.CVcited by 42

Not on the map. Why?

Abstract (arXiv)

Contrastive Language-Image Pre-Training (CLIP) is highly instrumental in machine learning applications within a large variety of domains. We investigate the geometry of this embedding, which is still not well understood. We examine the raw unnormalized embedding and show that text and image reside on linearly separable ellipsoid shells, not centered at the origin. We explain the benefits of having this structure, allowing to better embed instances according to their uncertainty during contrastive training. Frequent concepts in the dataset yield more false negatives, inducing greater uncertainty. A new notion of conformity is introduced, which measures the average cosine similarity of an instance to any other instance within a representative data set. We show this measure can be accurately estimated by simply computing the cosine similarity to the modality mean vector. Furthermore, we find that CLIP's modality gap optimizes the matching of the conformity distributions of image and text.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 47 of its 54 references that are in the map. First posted on arXiv 21 Nov 2024.

Builds on

47 of 54 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  8. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  11. 2019Natural Adversarial Examplescited by 1,401
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
35 more
  1. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  2. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  3. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  4. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  5. 2020Contrastive Learning with Hard Negative Samplescited by 619
  6. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  7. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  8. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  9. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  10. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  11. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  12. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  13. 2020Debiased Contrastive Learningcited by 354
  14. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  15. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  16. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
  17. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  18. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  19. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  20. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  21. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  22. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  23. 2021CLIP4Caption: CLIP for Video Captioncited by 88
  24. 2023Improving Tuning-Free Real Image Editing with Proximal Guidancecited by 87
  25. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  26. 2022Fine-grained Image Captioning with CLIP Rewardcited by 69
  27. 2021The Power of Contrast for Feature Learning: A Theoretical Analysiscited by 53
  28. 2023T-MARS: Improving Visual Representations by Circumventing Text Feature Learningcited by 43
  29. 2022ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCOcited by 37
  30. 2022Open-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfercited by 33
  31. 2023Too Large; Data Reduction for Vision-Language Pre-Trainingcited by 26
  32. 2024It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gapcited by 26
  33. 2022GRIT-VLP: Grouped Mini-batch Sampling for Efficient Vision and Language Pre-trainingcited by 10
  34. 2022A Slightly Improved Bound for the KLS Constantcited by 10
  35. 2023MAFA: Managing False Negatives for Vision-Language Pre-trainingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 31.

42 papers.

Sort:

By area:

  1. 2025PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selectioncited by 57
  2. 2025Is CLIP ideal? No. Can we fix it? Yes!cited by 20
  3. 2025Whitened CLIP as a Likelihood Surrogate of Images and Captionscited by 16
  4. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  5. 2024Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddingscited by 15
  6. 2026InfoNCE Induces Gaussian Distributioncited by 14
  7. 2026Cross-Modal Redundancy and the Geometry of Vision-Language Embeddingscited by 9
  8. 2026Training-free Detection of Generated Videos via Spatial-Temporal Likelihoodscited by 7
  9. 2025Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learningcited by 4
  10. 2025On the rankability of visual embeddingscited by 4
  11. 2025On the Brittleness of CLIP Text Encoderscited by 3
  12. 2026IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignmentcited by 3
30 more
  1. 2026The Universal Normal Embeddingcited by 3
  2. 2025Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Modelscited by 2
  3. 2026Epistemic Uncertainty Quantification for Pre-trained VLMs via Riemannian Flow Matchingcited by 2
  4. 2026TextME: Bridging Unseen Modalities Through Text Descriptionscited by 2
  5. 2026Global Geometry Is Not Enough for Vision Representationscited by 2
  6. 2026Cross-Modal Prototype Alignment and Mixing for Training-Free Few-Shot Classificationcited by 2
  7. 2025Personalized Image Generation for Recommendations Beyond Catalogscited by 1
  8. 2026Time Series, Vision, and Language: Exploring the Limits of Alignment in Contrastive Representation Spacescited by 1
  9. 2026SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysiscited by 1
  10. 2026Consistency Regularised Gradient Flows for Inverse Problemscited by 1
  11. 2026When to Align, When to Predict: A Phase Diagram for Multimodal Learningcited by 1
  12. 2025CLIP-like Model as a Foundational Density Ratio Estimatorcited by 0
  13. 2026Beyond Pooling: Matching for Robust Generalization under Data Heterogeneitycited by 0
  14. 2026Make it SING: Analyzing Semantic Invariants in Classifierscited by 0
  15. 2026HyCal: A Training-Free Prototype Calibration Method for Cross-Discipline Few-Shot Class-Incremental Learningcited by 0
  16. 2026HEART: Hyperspherical Embedding Alignment via Kent-Representation Traversal in Diffusion Modelscited by 0
  17. 2026GeoFlowVLM: Geometry-Aware Joint Uncertainty for Frozen Vision-Language Embeddingcited by 0
  18. 2026Density-Aware Translation of Spurious Correlations in Zero-Shot VLMscited by 0
  19. 2026Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weightingcited by 0
  20. 2026Beyond Templates: Revisiting Zero-Shot Remote Sensing through Meta-Promptingcited by 0
  21. 2026A Classifier-Agnostic Zero-Shot Adversarial Attack Detection via CLIPcited by 0
  22. 2026Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Normscited by 0
  23. 2026On the modality gap and the contrastive loss in multi-modal representation learningcited by 0
  24. 2026The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Modelcited by 0
  25. 2026GeoDetect: Geometric Adversarial Detection for VLPscited by 0
  26. 2026dRAE: Representation Autoencoder with Hyper-Spherical Codescited by 0
  27. 2026D'ej`a Cue: Localizing States in Object Histories via Vocabulary-Relative Coordinatescited by 0
  28. 2026Whitening Inverts the Hierarchy: What the Norm of a Whitened Embedding Measurescited by 0
  29. 2026One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Modelscited by 0
  30. 2026Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoderscited by 0