# Is Cosine-Similarity of Embeddings Really About Similarity?

arXiv: https://arxiv.org/abs/2403.05440 · HTML: https://mlsanity.com/paper/2403.05440/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 7 references that are in the map.
- First posted on arXiv 8 Mar 2024.

## Builds on (most cited first, 6 of 6)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words](https://mlsanity.com/paper/2205.05092/) (2022)
- [Towards a Better Understanding of Linear Models for Recommendation](https://mlsanity.com/paper/2105.12937/) (2021)
- [Regularized Singular Value Decomposition and Application to Recommender System](https://mlsanity.com/paper/1804.05090/) (2018)

## Built on by (most cited first, 30 of 89)

- [TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos](https://mlsanity.com/paper/2504.17343/) (2025)
- [Do Multilingual LLMs Think In English?](https://mlsanity.com/paper/2502.15603/) (2025)
- [Who's asking? User personas and the mechanics of latent misalignment](https://mlsanity.com/paper/2406.12094/) (2024)
- [Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages](https://mlsanity.com/paper/2404.11553/) (2024)
- [What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration](https://mlsanity.com/paper/2410.20482/) (2024)
- [Logically Consistent Language Models via Neuro-Symbolic Integration](https://mlsanity.com/paper/2409.13724/) (2024)
- [Optimal Design for Reward Modeling in RLHF](https://mlsanity.com/paper/2410.17055/) (2024)
- [Is CLIP ideal? No. Can we fix it? Yes!](https://mlsanity.com/paper/2503.08723/) (2025)
- [Evaluation of RAG Metrics for Question Answering in the Telecom Domain](https://mlsanity.com/paper/2407.12873/) (2024)
- [Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark](https://mlsanity.com/paper/2406.01607/) (2024)
- [Semantics at an Angle: When Cosine Similarity Works Until It Doesn't](https://mlsanity.com/paper/2504.16318/) (2025)
- [Data Generation Using Large Language Models for Text Classification: An Empirical Case Study](https://mlsanity.com/paper/2407.12813/) (2024)
- [Token-based Decision Criteria Are Suboptimal in In-context Learning](https://mlsanity.com/paper/2406.16535/) (2024)
- [DataLab: A Unified Platform for LLM-Powered Business Intelligence](https://mlsanity.com/paper/2412.02205/) (2024)
- [IM-Context: In-Context Learning for Imbalanced Regression Tasks](https://mlsanity.com/paper/2405.18202/) (2024)
- [Principled Context Engineering for RAG: Statistical Guarantees via Conformal Prediction](https://mlsanity.com/paper/2511.17908/) (2025)
- [Revisiting Cosine Similarity via Normalized ICA-transformed Embeddings](https://mlsanity.com/paper/2406.10984/) (2024)
- [Enhancing LLM-Based Text Classification in Political Science: Automatic Prompt Optimization and Dynamic Exemplar Selection for Few-Shot Learning](https://mlsanity.com/paper/2409.01466/) (2024)
- [Preserving Privacy and Utility in LLM-Based Product Recommendations](https://mlsanity.com/paper/2505.00951/) (2025)
- [Network-Level Prompt and Trait Leakage in Local Research Agents](https://mlsanity.com/paper/2508.20282/) (2025)
- [COS-Mix: Cosine Similarity and Distance Fusion for Improved Information Retrieval](https://mlsanity.com/paper/2406.00638/) (2024)
- [Quantifying Positional Biases in Text Embedding Models](https://mlsanity.com/paper/2412.15241/) (2024)
- [A novel Facial Recognition technique with Focusing on Masked Faces](https://mlsanity.com/paper/2501.04444/) (2025)
- [Modeling Hierarchical Spaces: A Review and Unified Framework for Surrogate-Based Architecture Design](https://mlsanity.com/paper/2506.22621/) (2025)
- [GENIUS: An Agentic AI Framework for Autonomous Design and Execution of Simulation Protocols](https://mlsanity.com/paper/2512.06404/) (2025)
- [IRSC: A Zero-shot Evaluation Benchmark for Information Retrieval through Semantic Comprehension in Retrieval-Augmented Generation Scenarios](https://mlsanity.com/paper/2409.15763/) (2024)
- [DoPAMine: Domain-specific Pre-training Adaptation from seed-guided data Mining](https://mlsanity.com/paper/2410.00260/) (2024)
- [This looks like what? Challenges and Future Research Directions for Part-Prototype Models](https://mlsanity.com/paper/2502.09340/) (2025)
- [ARCeR: an Agentic RAG for the Automated Definition of Cyber Ranges](https://mlsanity.com/paper/2504.12143/) (2025)
- [TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detection](https://mlsanity.com/paper/2504.17902/) (2025)

## Abstract (arXiv)

Cosine-similarity is the cosine of the angle between two vectors, or equivalently the dot product between their normalizations. A popular application is to quantify semantic similarity between high-dimensional objects by applying cosine-similarity to a learned low-dimensional feature embedding. This can work better but sometimes also worse than the unnormalized dot-product between embedded vectors in practice. To gain insight into this empirical observation, we study embeddings derived from regularized linear models, where closed-form solutions facilitate analytical insights. We derive analytically how cosine-similarity can yield arbitrary and therefore meaningless `similarities.' For some linear models the similarities are not even unique, while for others they are implicitly controlled by the regularization. We discuss implications beyond linear models: a combination of different regularizations are employed when learning deep models; these have implicit and unintended effects when taking cosine-similarities of the resulting embeddings, rendering results opaque and possibly arbitrary. Based on these insights, we caution against blindly using cosine-similarity and outline alternatives.