mlsanityData to 14 Oct 2026

Is Cosine-Similarity of Embeddings Really About Similarity?

arXiv 2403.05440 ↗First posted 8 Mar 2024cs.IR cs.LGcited by 89

Not on the map. Why?

Abstract (arXiv)

Cosine-similarity is the cosine of the angle between two vectors, or equivalently the dot product between their normalizations. A popular application is to quantify semantic similarity between high-dimensional objects by applying cosine-similarity to a learned low-dimensional feature embedding. This can work better but sometimes also worse than the unnormalized dot-product between embedded vectors in practice. To gain insight into this empirical observation, we study embeddings derived from regularized linear models, where closed-form solutions facilitate analytical insights. We derive analytically how cosine-similarity can yield arbitrary and therefore meaningless `similarities.' For some linear models the similarities are not even unique, while for others they are implicitly controlled by the regularization. We discuss implications beyond linear models: a combination of different regularizations are employed when learning deep models; these have implicit and unintended effects when taking cosine-similarities of the resulting embeddings, rendering results opaque and possibly arbitrary. Based on these insights, we caution against blindly using cosine-similarity and outline alternatives.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Builds on 6 of its 7 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

6 of 7 references.

References · 6

  1. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  3. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  4. 2022Problems with Cosine as a Measure of Embedding Similarity for High Frequency Wordscited by 49
  5. 2021Towards a Better Understanding of Linear Models for Recommendationcited by 18
  6. 2018Regularized Singular Value Decomposition and Application to Recommender Systemcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 25; 2025: 34; 2026: 30.

89 papers.

Sort:

By area:

  1. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 97
  2. 2025Do Multilingual LLMs Think In English?cited by 78
  3. 2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
  4. 2024Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languagescited by 38
  5. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  6. 2024Logically Consistent Language Models via Neuro-Symbolic Integrationcited by 30
  7. 2024Optimal Design for Reward Modeling in RLHFcited by 25
  8. 2025Is CLIP ideal? No. Can we fix it? Yes!cited by 21
  9. 2024Evaluation of RAG Metrics for Question Answering in the Telecom Domaincited by 20
  10. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmarkcited by 19
  11. 2025Semantics at an Angle: When Cosine Similarity Works Until It Doesn'tcited by 13
  12. 2024Data Generation Using Large Language Models for Text Classification: An Empirical Case Studycited by 11
38 more
  1. 2024Token-based Decision Criteria Are Suboptimal in In-context Learningcited by 9
  2. 2024DataLab: A Unified Platform for LLM-Powered Business Intelligencecited by 8
  3. 2024IM-Context: In-Context Learning for Imbalanced Regression Taskscited by 7
  4. 2025Principled Context Engineering for RAG: Statistical Guarantees via Conformal Predictioncited by 6
  5. 2024Revisiting Cosine Similarity via Normalized ICA-transformed Embeddingscited by 5
  6. 2024Enhancing LLM-Based Text Classification in Political Science: Automatic Prompt Optimization and Dynamic Exemplar Selection for Few-Shot Learningcited by 5
  7. 2025Preserving Privacy and Utility in LLM-Based Product Recommendationscited by 5
  8. 2025Network-Level Prompt and Trait Leakage in Local Research Agentscited by 5
  9. 2024COS-Mix: Cosine Similarity and Distance Fusion for Improved Information Retrievalcited by 4
  10. 2024Quantifying Positional Biases in Text Embedding Modelscited by 4
  11. 2025A novel Facial Recognition technique with Focusing on Masked Facescited by 4
  12. 2025Modeling Hierarchical Spaces: A Review and Unified Framework for Surrogate-Based Architecture Designcited by 4
  13. 2025GENIUS: An Agentic AI Framework for Autonomous Design and Execution of Simulation Protocolscited by 4
  14. 2024IRSC: A Zero-shot Evaluation Benchmark for Information Retrieval through Semantic Comprehension in Retrieval-Augmented Generation Scenarioscited by 3
  15. 2024DoPAMine: Domain-specific Pre-training Adaptation from seed-guided data Miningcited by 3
  16. 2025This looks like what? Challenges and Future Research Directions for Part-Prototype Modelscited by 3
  17. 2025ARCeR: an Agentic RAG for the Automated Definition of Cyber Rangescited by 3
  18. 2025TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detectioncited by 3
  19. 2025Contextual Representation Flow Across Depth in State-Space and Transformer Architecturescited by 3
  20. 2025SocialDF: Benchmark Dataset and Detection Model for Mitigating Harmful Deepfake Content on Social Media Platformscited by 2
  21. 2025Exploring LLM Capabilities in Extracting DCAT-Compatible Metadata for Data Catalogingcited by 2
  22. 2025Event Causality Identification with Synthetic Controlcited by 2
  23. 2025Demystifying Network Foundation Modelscited by 2
  24. 2026ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformerscited by 2
  25. 2024Implementing NLPs in industrial process modeling: Addressing Categorical Variablescited by 1
  26. 2024SimO Loss: Anchor-Free Contrastive Loss for Fine-Grained Supervised Contrastive Learningcited by 1
  27. 2025Improving Network Threat Detection by Knowledge Graph, Large Language Model, and Imbalanced Learningcited by 1
  28. 2025Neural Network Graph Similarity Computation Based on Graph Fusioncited by 1
  29. 2025Attacks and Defenses Against LLM Fingerprintingcited by 1
  30. 2025Do LLMs exhibit the same commonsense capabilities across languages?cited by 1
  31. 2025How Small Transformation Expose the Weakness of Semantic Similarity Measurescited by 1
  32. 2025DA-Font: Few-Shot Font Generation via Dual-Attention Hybrid Integrationcited by 1
  33. 2025Explainable AI for Analyzing Person-Specific Patterns in Facial Recognition Taskscited by 1
  34. 2025Multiple Treatments Causal Effects Estimation with Task Embeddings and Balanced Representation Learningcited by 1
  35. 2026Beyond Cosine Similaritycited by 1
  36. 2026Gauge Freedom and Metric Dependence in Neural Representation Spacescited by 1
  37. 2026Perturbation: A simple and efficient adversarial tracer for representation learning in language modelscited by 1
  38. 2026What Cosine Similarity of Label Representations Can and Cannot Tell uscited by 1

50 most cited of 89 All 89 (JSON)