mlsanityData to 14 Oct 2026

Revisiting Cosine Similarity via Normalized ICA-transformed Embeddings

arXiv 2406.10984 ↗First posted 16 Jun 2024cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Cosine similarity is widely used to measure the similarity between two embeddings, while interpretations based on angle and correlation coefficient are common. In this study, we focus on the interpretable axes of embeddings transformed by Independent Component Analysis (ICA), and propose a novel interpretation of cosine similarity as the sum of semantic similarities over axes. The normalized ICA-transformed embeddings exhibit sparsity, enhancing the interpretability of each axis, and the semantic similarity defined by the product of the components represents the shared meaning between the two embeddings along each axis. The effectiveness of this approach is demonstrated through intuitive numerical examples and thorough numerical experiments. By deriving the probability distributions that govern each component and the product of components, we propose a method for selecting statistically significant axes.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 32 of its 72 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

32 of 72 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  9. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  10. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  11. 2017Word Translation Without Parallel Datacited by 487
  12. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
20 more
  1. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 202
  2. 2013Exploiting Similarities among Languages for Machine Translationcited by 191
  3. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 172
  4. 2018Gromov-Wasserstein Alignment of Word Embedding Spacescited by 122
  5. 2017Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Featurescited by 104
  6. 2024Is Cosine-Similarity of Embeddings Really About Similarity?cited by 89
  7. 2017SPINE: SParse Interpretable Neural Embeddingscited by 55
  8. 2015Sparse Overcomplete Word Vector Representationscited by 44
  9. 2020Word Rotator's Distancecited by 32
  10. 2021Learning to Remove: Towards Isotropic Pre-trained BERT Embeddingcited by 23
  11. 2015Measuring Word Significance using Distributed Representations of Wordscited by 21
  12. 2022Norm of Word Embedding Encodes Information Gaincited by 21
  13. 2023Discovering Universal Geometry in Embeddings with ICAcited by 19
  14. 2017How to evaluate word embeddings? On importance of data efficiency and simple supervised taskscited by 12
  15. 2019A Comparison of Semantic Similarity Methods for Maximum Human Interpretabilitycited by 9
  16. 2018How much does a word weigh? Weighting word embeddings for word sense inductioncited by 6
  17. 2024Axis Tour: Word Tour Determines the Order of Axes in ICA-transformed Embeddingscited by 6
  18. 2023Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Wordscited by 5
  19. 2019Examining Structure of Word Embeddings with PCAcited by 3
  20. 2020Reflection-based Word Attribute Transfercited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2025Safe Pruning LoRA: Robust Distance-Guided Pruning for Safety Alignment in Adaptation of LLMscited by 10
  2. 2026Measuring Affinity between Attention-Head Weight Subspaces via the Projection Kernelcited by 5
  3. 2024Norm of Mean Contextualized Embeddings Determines their Variancecited by 1
  4. 2024Transforming Hidden States into Binary Semantic Featurescited by 0
  5. 2025Sharp Eyes and Memory for VideoLLMs: Information-Aware Visual Token Pruning for Efficient and Reliable VideoLLM Reasoningcited by 0