# Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora

arXiv: https://arxiv.org/abs/2112.14330 · HTML: https://mlsanity.com/paper/2112.14330/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 46 references that are in the map.
- First posted on arXiv 28 Dec 2021.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018)
- [Diachronic word embeddings and semantic shifts: a survey](https://mlsanity.com/paper/1806.03537/) (2018)
- [Statistically Significant Detection of Linguistic Change](https://mlsanity.com/paper/1411.3315/) (2014)
- [Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change](https://mlsanity.com/paper/1606.02821/) (2016)
- [Survey of Computational Approaches to Lexical Semantic Change](https://mlsanity.com/paper/1811.06278/) (2018)
- [Temporal Analysis of Language through Neural Language Models](https://mlsanity.com/paper/1405.3515/) (2014)
- [Dynamic Word Embeddings](https://mlsanity.com/paper/1702.08359/) (2017)
- [Dynamic Word Embeddings for Evolving Semantic Discovery](https://mlsanity.com/paper/1703.00607/) (2017)
- [Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shift](https://mlsanity.com/paper/1912.01072/) (2019)
- [Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change](https://mlsanity.com/paper/1804.06517/) (2018)
- [Factors Influencing the Surprising Instability of Word Embeddings](https://mlsanity.com/paper/1804.09692/) (2018)
- [A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains](https://mlsanity.com/paper/1906.02979/) (2019)
- [Short-Term Meaning Shift: A Distributional Exploration](https://mlsanity.com/paper/1809.03169/) (2018)
- [That's sick dude!: Automatic identification of word sense change across different timescales](https://mlsanity.com/paper/1405.4392/) (2014)
- [Socioeconomic Dependencies of Linguistic Patterns in Twitter: A Multivariate Analysis](https://mlsanity.com/paper/1804.01155/) (2018)
- [Structured Embedding Models for Grouped Data](https://mlsanity.com/paper/1709.10367/) (2017)
- [Generating Timelines by Modeling Semantic Change](https://mlsanity.com/paper/1909.09907/) (2019)

## Built on by (most cited first, 21 of 21)

- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Time is Encoded in the Weights of Finetuned Language Models](https://mlsanity.com/paper/2312.13401/) (2023)
- [A Survey on Contextualised Semantic Shift Detection](https://mlsanity.com/paper/2304.01666/) (2023)
- [Temporal Attention for Language Models](https://mlsanity.com/paper/2202.02093/) (2022)
- [Discovering Differences in the Representation of People using Contextualized Semantic Axes](https://mlsanity.com/paper/2210.12170/) (2022)
- [Graph-based Clustering for Detecting Semantic Change Across Time and Languages](https://mlsanity.com/paper/2402.01025/) (2024)
- [Substitution-based Semantic Change Detection using Contextual Embeddings](https://mlsanity.com/paper/2309.02403/) (2023)
- [An Assessment of the Impact of OCR Noise on Language Models](https://mlsanity.com/paper/2202.00470/) (2022)
- [A Greek Parliament Proceedings Dataset for Computational Linguistics and Political Analysis](https://mlsanity.com/paper/2210.12883/) (2022)
- [Measuring Intersectional Biases in Historical Documents](https://mlsanity.com/paper/2305.12376/) (2023)
- [Towards Debiasing Translation Artifacts](https://mlsanity.com/paper/2205.08001/) (2022)
- [The Problem of Semantic Shift in Longitudinal Monitoring of Social Media: A Case Study on Mental Health During the COVID-19 Pandemic](https://mlsanity.com/paper/2206.11160/) (2022)
- [Variation and Instability in Dialect-Based Embedding Spaces](https://mlsanity.com/paper/2303.14963/) (2023)
- [Presence or Absence: Are Unknown Word Usages in Dictionaries?](https://mlsanity.com/paper/2406.00656/) (2024)
- [Tracking Semantic Change in Slovene: A Novel Dataset and Optimal Transport-Based Distance](https://mlsanity.com/paper/2402.16596/) (2024)
- [A Systematic Survey of Natural Language Processing for the Greek Language](https://mlsanity.com/paper/2407.09861/) (2024)
- [Improving Interpretability of Lexical Semantic Change with Neurobiological Features](https://mlsanity.com/paper/2602.09760/) (2026)
- [Analysing the Greek Parliament Records with Emotion Classification](https://mlsanity.com/paper/2205.12012/) (2022)
- [Domain-Specific Word Embeddings with Structure Prediction](https://mlsanity.com/paper/2210.04962/) (2022)
- [Contextualized Word Vector-based Methods for Discovering Semantic Differences with No Training nor Word Alignment](https://mlsanity.com/paper/2305.11516/) (2023)
- [Measuring the Semantic Structure and Evolution of Conspiracy Theories](https://mlsanity.com/paper/2603.26062/) (2026)

## Abstract (arXiv)

The problem of comparing two bodies of text and searching for words that differ in their usage between them arises often in digital humanities and computational social science. This is commonly approached by training word embeddings on each corpus, aligning the vector spaces, and looking for words whose cosine distance in the aligned space is large. However, these methods often require extensive filtering of the vocabulary to perform well, and - as we show in this work - result in unstable, and hence less reliable, results. We propose an alternative approach that does not use vector space alignment, and instead considers the neighbors of each word. The method is simple, interpretable and stable. We demonstrate its effectiveness in 9 different setups, considering different corpus splitting criteria (age, gender and profession of tweet authors, time of tweet) and different languages (English, French and Hebrew).