mlsanityData to 7 Oct 2026

Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words

arXiv 2205.05092 ↗First posted 10 May 2022cs.CL cs.AIcited by 49

Not on the map. Why?

Abstract (arXiv)

Cosine similarity of contextual embeddings is used in many NLP tasks (e.g., QA, IR, MT) and metrics (e.g., BERTScore). Here, we uncover systematic ways in which word similarities estimated by cosine over BERT embeddings are understated and trace this effect to training data frequency. We find that relative to human judgements, cosine similarity underestimates the similarity of frequent words with other instances of the same word or other words across contexts, even after controlling for polysemy and other factors. We conjecture that this underestimation of similarity for high frequency words is due to differences in the representational geometry of high and low frequency words and provide a formal argument for the two-dimensional case.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 10 May 2022.

Builds on

23 of 32 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2018Datasheets for Datasetscited by 1,371
  4. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  5. 2018Model Cards for Model Reportingcited by 1,068
  6. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  7. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  8. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  9. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  10. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  11. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  12. 2019Visualizing and Measuring the Geometry of BERTcited by 215
11 more
  1. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  2. 2020On the Sentence Embeddings from Pre-trained Language Modelscited by 83
  3. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  4. 2020Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoderscited by 61
  5. 2019Understanding Undesirable Word Embedding Associationscited by 59
  6. 2018Towards Understanding Linear Word Analogiescited by 57
  7. 2021Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarkingcited by 53
  8. 2018Factors Influencing the Surprising Instability of Word Embeddingscited by 31
  9. 2020Utility is in the Eye of the User: A Critique of NLP Leaderboardscited by 28
  10. 2021Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metriccited by 21
  11. 2021BERT Has Uncommon Sense: Similarity Ranking for Word Sense BERTologycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 12; 2024: 17; 2025: 9; 2026: 9.

49 papers.

Sort:

By area:

  1. 2024Is Cosine-Similarity of Embeddings Really About Similarity?cited by 89
  2. 2024Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMcited by 77
  3. 2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
  4. 2025Language-Specific Latent Process Hinders Cross-Lingual Performancecited by 20
  5. 2025Semantics at an Angle: When Cosine Similarity Works Until It Doesn'tcited by 13
  6. 2023What Do Llamas Really Think? Revealing Preference Biases in Language Model Representationscited by 10
  7. 2025Understanding LLM Scientific Reasoning through Promptings and Model's Explanation on the Answerscited by 10
  8. 2023Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performancecited by 9
  9. 2024A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why?cited by 9
  10. 2023Unsupervised Semantic Variation Prediction using the Distribution of Sibling Embeddingscited by 8
  11. 2023Using Artificial Populations to Study Psychological Phenomena in Neural Modelscited by 8
  12. 2023A Reliable Knowledge Processing Framework for Combustion Science using Foundation Modelscited by 7
37 more
  1. 2025DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understandingcited by 7
  2. 2023Solving Cosine Similarity Underestimation between High Frequency Words by L2 Norm Discountingcited by 5
  3. 2023Translating away Translationese without Parallel Datacited by 5
  4. 2024AutoBreach: Universal and Adaptive Jailbreaking with Efficient Wordplay-Guided Optimizationcited by 5
  5. 2024Pragmatic inference of scalar implicature by LLMscited by 5
  6. 2022Word Embedding for Social Sciences: An Interdisciplinary Surveycited by 3
  7. 2024Semantics or spelling? Probing contextual word embeddings with orthographic noisecited by 3
  8. 2024IRSC: A Zero-shot Evaluation Benchmark for Information Retrieval through Semantic Comprehension in Retrieval-Augmented Generation Scenarioscited by 3
  9. 2023Unsupervised Sentence Representation Learning with Frequency-induced Adversarial Tuning and Incomplete Sentence Filteringcited by 2
  10. 2023The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translationcited by 2
  11. 2025Rethinking Word Similarity: Semantic Similarity through Classification Confusioncited by 2
  12. 2025Tversky Neural Networks: Psychologically Plausible Deep Learning with Differentiable Tversky Similaritycited by 2
  13. 2022Dictionary-Assisted Supervised Contrastive Learningcited by 1
  14. 2023Labels Need Prompts Too: Mask Matching for Natural Language Understanding Taskscited by 1
  15. 2024Injecting Wiktionary to improve token-level contextual representations using contrastive learningcited by 1
  16. 2024When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementalitycited by 1
  17. 2024Investigating Distributions of Telecom Adapted Sentence Embeddings for Document Retrievalcited by 1
  18. 2024Leveraging a Cognitive Model to Measure Subjective Similarity of Human and GPT-4 Written Contentcited by 1
  19. 2024Norm of Mean Contextualized Embeddings Determines their Variancecited by 1
  20. 2024Detecting Bias and Enhancing Diagnostic Accuracy in Large Language Models for Healthcarecited by 1
  21. 2025Data-Efficient Biomedical In-Context Learning: A Diversity-Enhanced Submodular Perspectivecited by 1
  22. 2026Beyond Cosine Similaritycited by 1
  23. 2026Perturbation: A simple and efficient adversarial tracer for representation learning in language modelscited by 1
  24. 2026Retrieval-Augmented Question Answering over Scientific Literature for the Electron-Ion Collidercited by 1
  25. 2026The Proxy Presumption: From Semantic Embeddings to Valid Social Measurescited by 1
  26. 2023Rethinking the Construction of Effective Metrics for Understanding the Mechanisms of Pretrained Language Modelscited by 0
  27. 2023Two-Stage Classifier for Campaign Negativity Detection using Axis Embeddings: A Case Study on Tweets of Political Users during 2021 Presidential Election in Irancited by 0
  28. 2024Topological quantification of ambiguity in semantic searchcited by 0
  29. 2024Benchmarking Floworks against OpenAI & Anthropic: A Novel Framework for Enhanced LLM Function Callingcited by 0
  30. 2024Embedding Space Allocation with Angle-Norm Joint Classifiers for Few-Shot Class-Incremental Learningcited by 0
  31. 2025Analyzing the Effect of Embedding Norms and Singular Values to Oversmoothing in Graph Neural Networkscited by 0
  32. 2025Breaking It Down: Domain-Aware Semantic Segmentation for Retrieval Augmented Generationcited by 0
  33. 2026Distance-to-Distance Ratio: A Similarity Measure for Sentences Based on Rate of Change in LLM Embeddingscited by 0
  34. 2026Embedding Learning on Multiplex Networks for Link Predictioncited by 0
  35. 2026Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Normscited by 0
  36. 2026Evaluating and Improving Pedagogical Fit in LLM-Based AI Tutors with the Pedagogical Suitability Indexcited by 0
  37. 2026Calibrating WEAT Against Anisotropy: ZCA Whitening as a Geometric Pre-Processing Step for Embedding Association Testscited by 0