# Rethinking Word Similarity: Semantic Similarity through Classification Confusion

arXiv: https://arxiv.org/abs/2502.05704 · HTML: https://mlsanity.com/paper/2502.05704/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 59 references that are in the map.
- First posted on arXiv 8 Feb 2025.

## Builds on (most cited first, 29 of 29)

- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [A Survey on Deep Learning for Named Entity Recognition](https://mlsanity.com/paper/1812.09449/) (2018)
- [Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes](https://mlsanity.com/paper/1711.08412/) (2017)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings](https://mlsanity.com/paper/1904.04047/) (2019)
- [Word Representations via Gaussian Embedding](https://mlsanity.com/paper/1412.6623/) (2014)
- [The Geometry of Culture: Analyzing Meaning through Word Embeddings](https://mlsanity.com/paper/1803.09288/) (2018)
- [Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change](https://mlsanity.com/paper/1606.02821/) (2016)
- [Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoders](https://mlsanity.com/paper/2005.02590/) (2020)
- [Discovering Latent Concepts Learned in BERT](https://mlsanity.com/paper/2205.07237/) (2022)
- [Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words](https://mlsanity.com/paper/2205.05092/) (2022)
- [Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shift](https://mlsanity.com/paper/1912.01072/) (2019)
- [NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets](https://mlsanity.com/paper/1308.6242/) (2013)
- [BERT Knows Punta Cana is not just beautiful, it's gorgeous: Ranking Scalar Adjectives with Contextualised Representations](https://mlsanity.com/paper/2010.02686/) (2020)
- [Slangvolution: A Causal Analysis of Semantic Change and Frequency Dynamics in Slang](https://mlsanity.com/paper/2203.04651/) (2022)
- [Discovering Differences in the Representation of People using Contextualized Semantic Axes](https://mlsanity.com/paper/2210.12170/) (2022)
- [Embedding Words as Distributions with a Bayesian Skip-gram Model](https://mlsanity.com/paper/1711.11027/) (2017)
- [A Comparison of Semantic Similarity Methods for Maximum Human Interpretability](https://mlsanity.com/paper/1910.09129/) (2019)
- [Methods for Computing Legal Document Similarity: A Comparative Study](https://mlsanity.com/paper/2004.12307/) (2020)
- [Representation Of Lexical Stylistic Features In Language Models' Embedding Space](https://mlsanity.com/paper/2305.18657/) (2023)
- [Adjusting Interpretable Dimensions in Embedding Space with Human Judgments](https://mlsanity.com/paper/2404.02619/) (2024)

## Built on by (most cited first, 2 of 2)

- [Express Your Doubts -- Probabilistic World Modeling Should not be Based on Token logprobs](https://mlsanity.com/paper/2505.02072/) (2025)
- [Relation Geometry in Semantic Space of Language Models](https://mlsanity.com/paper/2607.26762/) (2026)

## Abstract (arXiv)

Word similarity has many applications to social science and cultural analytics tasks like measuring meaning change over time and making sense of contested terms. Yet traditional similarity methods based on cosine similarity between word embeddings cannot capture the context-dependent, asymmetrical, polysemous nature of semantic similarity. We propose a new measure of similarity, Word Confusion, that reframes semantic similarity in terms of feature-based classification confusion. Word Confusion is inspired by Tversky's suggestion that similarity features be chosen dynamically. Here we train a classifier to map contextual embeddings to word identities and use the classifier confusion (the probability of choosing a confounding word c instead of the correct target word t) as a measure of the similarity of c and t. The set of potential confounding words acts as the chosen features. Our method is comparable to cosine similarity in matching human similarity judgments across several datasets (MEN, WirdSim353, and SimLex), and can measure similarity using predetermined features of interest. We demonstrate our model's ability to make use of dynamic features by applying it to test a hypothesis about changes in the 18th C. meaning of the French word "revolution" from popular to state action during the French Revolution. We hope this reimagining of semantic similarity will inspire the development of new tools that better capture the multi-faceted and dynamic nature of language, advancing the fields of computational social science and cultural analytics and beyond.