# Analysing Lexical Semantic Change with Contextualised Word Representations

arXiv: https://arxiv.org/abs/2004.14118 · HTML: https://mlsanity.com/paper/2004.14118/

## Key facts

- Cited by 92 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2020.

## Built on by (most cited first, 30 of 92)

- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [Time Masking for Temporal Language Models](https://mlsanity.com/paper/2110.06366/) (2021)
- [Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Senses](https://mlsanity.com/paper/2104.14694/) (2021)
- [Explaining and Improving BERT Performance on Lexical Semantic Change Detection](https://mlsanity.com/paper/2103.07259/) (2021)
- [LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models](https://mlsanity.com/paper/2402.02896/) (2024)
- [Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research](https://mlsanity.com/paper/2412.04497/) (2024)
- [DWUG: A large Resource of Diachronic Word Usage Graphs in Four Languages](https://mlsanity.com/paper/2104.08540/) (2021)
- [A Survey on Contextualised Semantic Shift Detection](https://mlsanity.com/paper/2304.01666/) (2023)
- [Temporal Attention for Language Models](https://mlsanity.com/paper/2202.02093/) (2022)
- [A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change](https://mlsanity.com/paper/2402.12011/) (2024)
- [Characterizing English Variation across Social Media Communities with BERT](https://mlsanity.com/paper/2102.06820/) (2021)
- [Contextualized language models for semantic change detection: lessons learned](https://mlsanity.com/paper/2209.00154/) (2022)
- [Causal Distillation for Language Models](https://mlsanity.com/paper/2112.02505/) (2021)
- [Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis](https://mlsanity.com/paper/2305.11993/) (2023)
- [Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change](https://mlsanity.com/paper/2210.17127/) (2022)
- [BiTimeBERT: Extending Pre-Trained Language Representations with Bi-Temporal Information](https://mlsanity.com/paper/2204.13032/) (2022)
- [Challenges for Computational Lexical Semantic Change](https://mlsanity.com/paper/2101.07668/) (2021)
- [Slangvolution: A Causal Analysis of Semantic Change and Frequency Dynamics in Slang](https://mlsanity.com/paper/2203.04651/) (2022)
- [TempRetriever: Fusion-based Temporal Dense Passage Retrieval for Time-Sensitive Questions](https://mlsanity.com/paper/2502.21024/) (2025)
- [Statistically significant detection of semantic shifts using contextual word embeddings](https://mlsanity.com/paper/2104.03776/) (2021)
- [On the Impact of Temporal Concept Drift on Model Explanations](https://mlsanity.com/paper/2210.09197/) (2022)
- [Lexical Semantic Change Discovery](https://mlsanity.com/paper/2106.03111/) (2021)
- [A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spaces](https://mlsanity.com/paper/2305.18598/) (2023)
- [Discovering Differences in the Representation of People using Contextualized Semantic Axes](https://mlsanity.com/paper/2210.12170/) (2022)
- [HistBERT: A Pre-trained Language Model for Diachronic Lexical Semantic Analysis](https://mlsanity.com/paper/2202.03612/) (2022)
- [Once Upon a textitTime in textitGraph: Relative-Time Pretraining for Complex Temporal Reasoning](https://mlsanity.com/paper/2310.14709/) (2023)
- [A Study into Investigating Temporal Robustness of LLMs](https://mlsanity.com/paper/2503.17073/) (2025)
- [Grammatical Profiling for Semantic Change Detection](https://mlsanity.com/paper/2109.10397/) (2021)
- [Can Word Sense Distribution Detect Semantic Changes of Words?](https://mlsanity.com/paper/2310.10400/) (2023)
- [Graph-based Clustering for Detecting Semantic Change Across Time and Languages](https://mlsanity.com/paper/2402.01025/) (2024)

## Abstract (arXiv)

This paper presents the first unsupervised approach to lexical semantic change that makes use of contextualised word representations. We propose a novel method that exploits the BERT neural language model to obtain representations of word usages, clusters these representations into usage types, and measures change along time with three proposed metrics. We create a new evaluation dataset and show that the model representations and the detected semantic shifts are positively correlated with human judgements. Our extensive qualitative analysis demonstrates that our method captures a variety of synchronic and diachronic linguistic phenomena. We expect our work to inspire further research in this direction.