mlsanityData to 7 Oct 2026

Embedding Geometries of Contrastive Language-Image Pre-Training

arXiv 2409.13079 ↗First posted 19 Sep 2024cs.LG cs.CL cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Since the publication of CLIP, the approach of using InfoNCE loss for contrastive pre-training has become widely popular for bridging two or more modalities. Despite its wide adoption, CLIP's original design choices of L2 normalization and cosine similarity logit have rarely been revisited. We have systematically experimented with alternative geometries and softmax logits for language-image pre-training and identified that variants with intuitive Euclidean geometry, Euclidean CLIP (EuCLIP), match or exceed the performance of CLIP and support hierarchical relationships at least as well as more complicated hyperbolic alternative.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 30 of its 34 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

30 of 34 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  9. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
18 more
  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  2. 2020On Layer Normalization in the Transformer Architecturecited by 906
  3. 2017Poincar'e Embeddings for Learning Hierarchical Representationscited by 674
  4. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  5. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  6. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  7. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  8. 2024The Faiss librarycited by 370
  9. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  10. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  11. 2020The Lipschitz Constant of Self-Attentioncited by 171
  12. 2018Hyperbolic Entailment Cones for Learning Hierarchical Embeddingscited by 160
  13. 2023Hyperbolic Image-Text Representationscited by 132
  14. 2021Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillationcited by 35
  15. 2023On the Expressivity Role of LayerNorm in Transformers' Attentioncited by 32
  16. 2022Hyperbolic Deep Reinforcement Learningcited by 25
  17. 2021Comparing Euclidean and Hyperbolic Embeddings on the WordNet Nouns Hypernymy Graphcited by 3
  18. 2023The Rise of Open Science: Tracking the Evolution and Perceived Value of Data and Methods Link-Sharing Practicescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Hyperbolic Safety-Aware Vision-Language Modelscited by 17
  2. 2025Whitened CLIP as a Likelihood Surrogate of Images and Captionscited by 16
  3. 2025NeuCLIP: Efficient Large-Scale CLIP Training with Neural Normalizer Optimizationcited by 3
  4. 2025Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similaritycited by 2
  5. 2026ARGENT: Adaptive Hierarchical Image-Text Representationscited by 1
  6. 2026Is the Geometry Doing the Work? An Operating-Point Audit of Hierarchy in Hyperbolic Vision-Language Modelscited by 0