# The Double-Ellipsoid Geometry of CLIP

arXiv: https://arxiv.org/abs/2411.14517 · HTML: https://mlsanity.com/paper/2411.14517/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 54 references that are in the map.
- First posted on arXiv 21 Nov 2024.

## Builds on (most cited first, 30 of 47)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding](https://mlsanity.com/paper/2007.10985/) (2020)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [Vision-Language Pre-Training with Triple Contrastive Learning](https://mlsanity.com/paper/2202.10401/) (2022)
- [Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP](https://mlsanity.com/paper/2308.02487/) (2023)
- [CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding](https://mlsanity.com/paper/2203.00680/) (2022)

## Built on by (most cited first, 30 of 42)

- [PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection](https://mlsanity.com/paper/2502.12119/) (2025)
- [Is CLIP ideal? No. Can we fix it? Yes!](https://mlsanity.com/paper/2503.08723/) (2025)
- [Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting](https://mlsanity.com/paper/2508.04227/) (2025)
- [Whitened CLIP as a Likelihood Surrogate of Images and Captions](https://mlsanity.com/paper/2505.06934/) (2025)
- [InfoNCE Induces Gaussian Distribution](https://mlsanity.com/paper/2602.24012/) (2026)
- [Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddings](https://mlsanity.com/paper/2412.18645/) (2024)
- [Cross-Modal Redundancy and the Geometry of Vision-Language Embeddings](https://mlsanity.com/paper/2602.06218/) (2026)
- [Training-free Detection of Generated Videos via Spatial-Temporal Likelihoods](https://mlsanity.com/paper/2603.15026/) (2026)
- [Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning](https://mlsanity.com/paper/2505.19614/) (2025)
- [On the rankability of visual embeddings](https://mlsanity.com/paper/2507.03683/) (2025)
- [On the Brittleness of CLIP Text Encoders](https://mlsanity.com/paper/2511.04247/) (2025)
- [IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment](https://mlsanity.com/paper/2603.19862/) (2026)
- [The Universal Normal Embedding](https://mlsanity.com/paper/2603.21786/) (2026)
- [Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Models](https://mlsanity.com/paper/2511.22019/) (2025)
- [Epistemic Uncertainty Quantification for Pre-trained VLMs via Riemannian Flow Matching](https://mlsanity.com/paper/2601.21662/) (2026)
- [TextME: Bridging Unseen Modalities Through Text Descriptions](https://mlsanity.com/paper/2602.03098/) (2026)
- [Global Geometry Is Not Enough for Vision Representations](https://mlsanity.com/paper/2602.03282/) (2026)
- [Time Series, Vision, and Language: Exploring the Limits of Alignment in Contrastive Representation Spaces](https://mlsanity.com/paper/2602.19367/) (2026)
- [Cross-Modal Prototype Alignment and Mixing for Training-Free Few-Shot Classification](https://mlsanity.com/paper/2603.24528/) (2026)
- [Consistency Regularised Gradient Flows for Inverse Problems](https://mlsanity.com/paper/2605.07907/) (2026)
- [Personalized Image Generation for Recommendations Beyond Catalogs](https://mlsanity.com/paper/2502.18477/) (2025)
- [SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysis](https://mlsanity.com/paper/2603.13884/) (2026)
- [HyCal: A Training-Free Prototype Calibration Method for Cross-Discipline Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2604.15678/) (2026)
- [When to Align, When to Predict: A Phase Diagram for Multimodal Learning](https://mlsanity.com/paper/2606.11190/) (2026)
- [CLIP-like Model as a Foundational Density Ratio Estimator](https://mlsanity.com/paper/2506.22881/) (2025)
- [Beyond Pooling: Matching for Robust Generalization under Data Heterogeneity](https://mlsanity.com/paper/2602.07154/) (2026)
- [Make it SING: Analyzing Semantic Invariants in Classifiers](https://mlsanity.com/paper/2603.14610/) (2026)
- [HEART: Hyperspherical Embedding Alignment via Kent-Representation Traversal in Diffusion Models](https://mlsanity.com/paper/2605.07973/) (2026)
- [GeoFlowVLM: Geometry-Aware Joint Uncertainty for Frozen Vision-Language Embedding](https://mlsanity.com/paper/2605.13352/) (2026)
- [Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs](https://mlsanity.com/paper/2606.01710/) (2026)

## Abstract (arXiv)

Contrastive Language-Image Pre-Training (CLIP) is highly instrumental in machine learning applications within a large variety of domains. We investigate the geometry of this embedding, which is still not well understood. We examine the raw unnormalized embedding and show that text and image reside on linearly separable ellipsoid shells, not centered at the origin. We explain the benefits of having this structure, allowing to better embed instances according to their uncertainty during contrastive training. Frequent concepts in the dataset yield more false negatives, inducing greater uncertainty. A new notion of conformity is introduced, which measures the average cosine similarity of an instance to any other instance within a representative data set. We show this measure can be accurately estimated by simply computing the cosine similarity to the modality mean vector. Furthermore, we find that CLIP's modality gap optimizes the matching of the conformity distributions of image and text.