mlsanityData to 14 Oct 2026

Semantics at an Angle: When Cosine Similarity Works Until It Doesn't

arXiv 2504.16318 ↗First posted 22 Apr 2025cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Cosine similarity is a standard comparison rule for learned representations in information retrieval, natural language processing, computer vision, and multimodal learning. Its popularity is well founded: it removes positive radial scale, is computationally convenient, and often matches objectives that train normalized embeddings. These same properties also delimit what cosine can express. Normalization discards radial variation; anisotropic representations can compress angular contrast; high-dimensional neighborhoods can develop hubs; and a symmetric, uncalibrated score may be mismatched to the relation of interest. This article offers a selective review organized around a simple principle: the usefulness of cosine similarity depends jointly on the learned representation, any normalization or post-processing, the scoring rule, and the downstream decision. We derive the main geometric identities, distinguish failure mechanisms that are often conflated, review representative evidence about embedding norms, and describe objective-matched, geometry-aware, hubness-aware, norm-aware, and learned alternatives. The central conclusion is conditional rather than adversarial: cosine's positive-scale invariance is justified when radial variation is nuisance or fixed by the representation contract, but its angular geometry and downstream decision must still be validated.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 22 Apr 2025.

Builds on

21 of 38 references.

References · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  12. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
9 more
  1. 2017Word Translation Without Parallel Datacited by 487
  2. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 212
  3. 2024Is Cosine-Similarity of Embeddings Really About Similarity?cited by 89
  4. 2021Cross Modal Retrieval with Querybank Normalisationcited by 72
  5. 2021WhiteningBERT: An Easy Unsupervised Sentence Embedding Approachcited by 69
  6. 2022Problems with Cosine as a Measure of Embedding Similarity for High Frequency Wordscited by 49
  7. 2020Word Rotator's Distancecited by 32
  8. 2022Norm of Word Embedding Encodes Information Gaincited by 21
  9. 2025On the Importance of Embedding Norms in Self-Supervised Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

By area: