# Statistically significant detection of semantic shifts using contextual word embeddings

arXiv: https://arxiv.org/abs/2104.03776 · HTML: https://mlsanity.com/paper/2104.03776/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 8 Apr 2021.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016)
- [SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection](https://mlsanity.com/paper/2007.11464/) (2020)
- [Diachronic word embeddings and semantic shifts: a survey](https://mlsanity.com/paper/1806.03537/) (2018)
- [Analysing Lexical Semantic Change with Contextualised Word Representations](https://mlsanity.com/paper/2004.14118/) (2020)
- [Statistically Significant Detection of Linguistic Change](https://mlsanity.com/paper/1411.3315/) (2014)
- [Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change](https://mlsanity.com/paper/1606.02821/) (2016)
- [Temporal Analysis of Language through Neural Language Models](https://mlsanity.com/paper/1405.3515/) (2014)
- [Latin BERT: A Contextual Language Model for Classical Philology](https://mlsanity.com/paper/2009.10053/) (2020)
- [Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shift](https://mlsanity.com/paper/1912.01072/) (2019)
- [Factors Influencing the Surprising Instability of Word Embeddings](https://mlsanity.com/paper/1804.09692/) (2018)
- [Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change](https://mlsanity.com/paper/1906.01688/) (2019)
- [UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection](https://mlsanity.com/paper/2005.00050/) (2020)
- [A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains](https://mlsanity.com/paper/1906.02979/) (2019)
- [Short-Term Meaning Shift: A Distributional Exploration](https://mlsanity.com/paper/1809.03169/) (2018)
- [That's sick dude!: Automatic identification of word sense change across different timescales](https://mlsanity.com/paper/1405.4392/) (2014)

## Built on by (most cited first, 14 of 14)

- [A Survey on Contextualised Semantic Shift Detection](https://mlsanity.com/paper/2304.01666/) (2023)
- [Substitution-based Semantic Change Detection using Contextual Embeddings](https://mlsanity.com/paper/2309.02403/) (2023)
- [textitSwap and Predict -- Predicting the Semantic Changes in Words across Corpora by Context Swapping](https://mlsanity.com/paper/2310.10397/) (2023)
- [A Greek Parliament Proceedings Dataset for Computational Linguistics and Political Analysis](https://mlsanity.com/paper/2210.12883/) (2022)
- [Tracing the Development of the Virtual Particle Concept Using Semantic Change Detection](https://mlsanity.com/paper/2410.16855/) (2024)
- [Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiC](https://mlsanity.com/paper/2211.03466/) (2022)
- [Large Language Models for History, Philosophy, and Sociology of Science: Interpretive Uses, Methodological Challenges, and Critical Perspectives](https://mlsanity.com/paper/2506.12242/) (2025)
- [Comfort Foods and Community Connectedness: Investigating Diet Change during COVID-19 Using YouTube Videos on Twitter](https://mlsanity.com/paper/2305.11398/) (2023)
- [Does a Large Language Model Really Speak in Human-Like Language?](https://mlsanity.com/paper/2501.01273/) (2025)
- [Improving Interpretability of Lexical Semantic Change with Neurobiological Features](https://mlsanity.com/paper/2602.09760/) (2026)
- [Text analysis and deep learning: A network approach](https://mlsanity.com/paper/2110.04151/) (2021)
- [Gamma Mixture Modeling for Cosine Similarity in Small Language Models](https://mlsanity.com/paper/2510.05309/) (2025)
- [Community-Specific Slang and Entity Detection via Semantic Shift in Fine-Tuned Language Models](https://mlsanity.com/paper/2606.07522/) (2026)
- [Beyond frequency measures: Can contextual embeddings capture meaning change in scientific texts?](https://mlsanity.com/paper/2609.18804/) (2026)

## Abstract (arXiv)

Detecting lexical semantic change in smaller data sets, e.g. in historical linguistics and digital humanities, is challenging due to a lack of statistical power. This issue is exacerbated by non-contextual embedding models that produce one embedding per word and, therefore, mask the variability present in the data. In this article, we propose an approach to estimate semantic shift by combining contextual word embeddings with permutation-based statistical tests. We use the false discovery rate procedure to address the large number of hypothesis tests being conducted simultaneously. We demonstrate the performance of this approach in simulation where it achieves consistently high precision by suppressing false positives. We additionally analyze real-world data from SemEval-2020 Task 1 and the Liverpool FC subreddit corpus. We show that by taking sample variation into account, we can improve the robustness of individual semantic shift estimates without degrading overall performance.