# Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shift

arXiv: https://arxiv.org/abs/1912.01072 · HTML: https://mlsanity.com/paper/1912.01072/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2019.

## Built on by (most cited first, 30 of 37)

- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [Time Masking for Temporal Language Models](https://mlsanity.com/paper/2110.06366/) (2021)
- [A Survey on Contextualised Semantic Shift Detection](https://mlsanity.com/paper/2304.01666/) (2023)
- [Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora](https://mlsanity.com/paper/2112.14330/) (2021)
- [Temporal Attention for Language Models](https://mlsanity.com/paper/2202.02093/) (2022)
- [A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change](https://mlsanity.com/paper/2402.12011/) (2024)
- [Contextualized language models for semantic change detection: lessons learned](https://mlsanity.com/paper/2209.00154/) (2022)
- [Statistically significant detection of semantic shifts using contextual word embeddings](https://mlsanity.com/paper/2104.03776/) (2021)
- [Discovering Differences in the Representation of People using Contextualized Semantic Axes](https://mlsanity.com/paper/2210.12170/) (2022)
- [Robustness of Large Language Models to Perturbations in Text](https://mlsanity.com/paper/2407.08989/) (2024)
- [Can Word Sense Distribution Detect Semantic Changes of Words?](https://mlsanity.com/paper/2310.10400/) (2023)
- [Substitution-based Semantic Change Detection using Contextual Embeddings](https://mlsanity.com/paper/2309.02403/) (2023)
- [What company do words keep? Revisiting the distributional semantics of J.R. Firth & Zellig Harris](https://mlsanity.com/paper/2205.07750/) (2022)
- [Unsupervised Semantic Variation Prediction using the Distribution of Sibling Embeddings](https://mlsanity.com/paper/2305.08654/) (2023)
- [Do Not Fire the Linguist: Grammatical Profiles Help Language Models Detect Semantic Change](https://mlsanity.com/paper/2204.05717/) (2022)
- [Time Machine GPT](https://mlsanity.com/paper/2404.18543/) (2024)
- [Analyzing Semantic Change through Lexical Replacements](https://mlsanity.com/paper/2404.18570/) (2024)
- [textitSwap and Predict -- Predicting the Semantic Changes in Words across Corpora by Context Swapping](https://mlsanity.com/paper/2310.10397/) (2023)
- [Social Meme-ing: Measuring Linguistic Variation in Memes](https://mlsanity.com/paper/2311.09130/) (2023)
- [A Semantic Distance Metric Learning approach for Lexical Semantic Change Detection](https://mlsanity.com/paper/2403.00226/) (2024)
- [Monitoring geometrical properties of word embeddings for detecting the emergence of new topics](https://mlsanity.com/paper/2111.03496/) (2021)
- [The LSCD Benchmark: a Testbed for Diachronic Word Meaning Tasks](https://mlsanity.com/paper/2404.00176/) (2024)
- [Tracing the Development of the Virtual Particle Concept Using Semantic Change Detection](https://mlsanity.com/paper/2410.16855/) (2024)
- [The challenges of temporal alignment on Twitter during crises](https://mlsanity.com/paper/2104.08535/) (2021)
- [When Dialects Collide: How Socioeconomic Mixing Affects Language Use](https://mlsanity.com/paper/2307.10016/) (2023)
- [Rethinking Word Similarity: Semantic Similarity through Classification Confusion](https://mlsanity.com/paper/2502.05704/) (2025)
- [Comparative Analysis of Static and Contextual Embeddings for Analyzing Semantic Changes in Medieval Latin Charters](https://mlsanity.com/paper/2410.09283/) (2024)
- [Identifying Emerging Concepts in Large Corpora](https://mlsanity.com/paper/2502.21315/) (2025)
- [Improving Interpretability of Lexical Semantic Change with Neurobiological Features](https://mlsanity.com/paper/2602.09760/) (2026)
- [DHPLT: large-scale multilingual diachronic corpora and word representations for semantic change modelling](https://mlsanity.com/paper/2602.11968/) (2026)

## Abstract (arXiv)

We propose a new method that leverages contextual embeddings for the task of diachronic semantic shift detection by generating time specific word representations from BERT embeddings. The results of our experiments in the domain specific LiverpoolFC corpus suggest that the proposed method has performance comparable to the current state-of-the-art without requiring any time consuming domain adaptation on large corpora. The results on the newly created Brexit news corpus suggest that the method can be successfully used for the detection of a short-term yearly semantic shift. And lastly, the model also shows promising results in a multilingual settings, where the task was to detect differences and similarities between diachronic semantic shifts in different languages.