# Contextualized language models for semantic change detection: lessons learned arXiv: https://arxiv.org/abs/2209.00154 ยท HTML: https://mlsanity.com/paper/2209.00154/ ## Key facts - Cited by 20 papers in the map (data through 14 Oct 2026). - Builds on 15 of its 36 references that are in the map. - First posted on arXiv 31 Aug 2022. ## Builds on (most cited first, 15 of 15) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013) - [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018) - [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016) - [Dissecting Contextual Word Embeddings: Architecture and Representation](https://mlsanity.com/paper/1808.08949/) (2018) - [SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection](https://mlsanity.com/paper/2007.11464/) (2020) - [Diachronic word embeddings and semantic shifts: a survey](https://mlsanity.com/paper/1806.03537/) (2018) - [Analysing Lexical Semantic Change with Contextualised Word Representations](https://mlsanity.com/paper/2004.14118/) (2020) - [Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change](https://mlsanity.com/paper/1606.02821/) (2016) - [Survey of Computational Approaches to Lexical Semantic Change](https://mlsanity.com/paper/1811.06278/) (2018) - [Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shift](https://mlsanity.com/paper/1912.01072/) (2019) - [UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection](https://mlsanity.com/paper/2005.00050/) (2020) - [Capturing Evolution in Word Usage: Just Add More Clusters?](https://mlsanity.com/paper/2001.06629/) (2020) - [Challenges for Computational Lexical Semantic Change](https://mlsanity.com/paper/2101.07668/) (2021) - [A State-of-the-Art of Semantic Change Computation](https://mlsanity.com/paper/1801.09872/) (2018) ## Built on by (most cited first, 20 of 20) - [A Survey on Contextualised Semantic Shift Detection](https://mlsanity.com/paper/2304.01666/) (2023) - [A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change](https://mlsanity.com/paper/2402.12011/) (2024) - [Can Word Sense Distribution Detect Semantic Changes of Words?](https://mlsanity.com/paper/2310.10400/) (2023) - [Graph-based Clustering for Detecting Semantic Change Across Time and Languages](https://mlsanity.com/paper/2402.01025/) (2024) - [Unsupervised Semantic Variation Prediction using the Distribution of Sibling Embeddings](https://mlsanity.com/paper/2305.08654/) (2023) - [Analyzing Semantic Change through Lexical Replacements](https://mlsanity.com/paper/2404.18570/) (2024) - [Definition generation for lexical semantic change detection](https://mlsanity.com/paper/2406.14167/) (2024) - [Evolving linguistic divergence on polarizing social media](https://mlsanity.com/paper/2309.01659/) (2023) - [Survey in Characterizing Semantic Change](https://mlsanity.com/paper/2402.19088/) (2024) - [A Multidimensional Framework for Evaluating Lexical Semantic Change with Social Science Applications](https://mlsanity.com/paper/2406.06052/) (2024) - [Large Language Models for History, Philosophy, and Sociology of Science: Interpretive Uses, Methodological Challenges, and Critical Perspectives](https://mlsanity.com/paper/2506.12242/) (2025) - [Enriching Word Usage Graphs with Cluster Definitions](https://mlsanity.com/paper/2403.18024/) (2024) - [Norm of Mean Contextualized Embeddings Determines their Variance](https://mlsanity.com/paper/2409.11253/) (2024) - [DHPLT: large-scale multilingual diachronic corpora and word representations for semantic change modelling](https://mlsanity.com/paper/2602.11968/) (2026) - [Evaluating the Evaluator: Problems with SemEval-2020 Task 1 for Lexical Semantic Change Detection](https://mlsanity.com/paper/2604.13232/) (2026) - [A Tale of Two Laws of Semantic Change: Predicting Synonym Changes with Distributional Semantic Models](https://mlsanity.com/paper/2305.19143/) (2023) - [Rethinking Metrics for Lexical Semantic Change Detection](https://mlsanity.com/paper/2602.15716/) (2026) - [Computational conceptual history of scientific concepts: From early digital methods to LLMs](https://mlsanity.com/paper/2606.04118/) (2026) - [Tracing individual knowledge trajectories in a changing field: the case of general relativity and gravitation](https://mlsanity.com/paper/2609.18697/) (2026) - [A Computational Framework for Modelling Organisation-Level Semantic Identity from Longitudinal Textual Data](https://mlsanity.com/paper/2609.29584/) (2026) ## Abstract (arXiv) We present a qualitative analysis of the (potentially erroneous) outputs of contextualized embedding-based methods for detecting diachronic semantic change. First, we introduce an ensemble method outperforming previously described contextualized approaches. This method is used as a basis for an in-depth analysis of the degrees of semantic change predicted for English words across 5 decades. Our findings show that contextualized methods can often predict high change scores for words which are not undergoing any real diachronic semantic shift in the lexicographic sense of the term (or at least the status of these shifts is questionable). Such challenging cases are discussed in detail with examples, and their linguistic categorization is proposed. Our conclusion is that pre-trained contextualized language models are prone to confound changes in lexicographic senses and changes in contextual variance, which naturally stem from their distributional nature, but is different from the types of issues observed in methods based on static embeddings. Additionally, they often merge together syntactic and semantic aspects of lexical entities. We propose a range of possible future solutions to these issues.