# Analyzing Semantic Change through Lexical Replacements

arXiv: https://arxiv.org/abs/2404.18570 · HTML: https://mlsanity.com/paper/2404.18570/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 52 references that are in the map.
- First posted on arXiv 29 Apr 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection](https://mlsanity.com/paper/2007.11464/) (2020)
- [SenseBERT: Driving Some Sense into BERT](https://mlsanity.com/paper/1908.05646/) (2019)
- [Survey of Computational Approaches to Lexical Semantic Change](https://mlsanity.com/paper/1811.06278/) (2018)
- [Time Masking for Temporal Language Models](https://mlsanity.com/paper/2110.06366/) (2021)
- [Why Attentions May Not Be Interpretable?](https://mlsanity.com/paper/2006.05656/) (2020)
- [Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Senses](https://mlsanity.com/paper/2104.14694/) (2021)
- [Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shift](https://mlsanity.com/paper/1912.01072/) (2019)
- [Explaining and Improving BERT Performance on Lexical Semantic Change Detection](https://mlsanity.com/paper/2103.07259/) (2021)
- [UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection](https://mlsanity.com/paper/2005.00050/) (2020)
- [Word Order Does Matter (And Shuffled Language Models Know It)](https://mlsanity.com/paper/2203.10995/) (2022)
- [A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change](https://mlsanity.com/paper/2402.12011/) (2024)
- [Contextualized language models for semantic change detection: lessons learned](https://mlsanity.com/paper/2209.00154/) (2022)
- [Learn over Past, Evolve for Future: Forecasting Temporal Trends for Fake News Detection](https://mlsanity.com/paper/2306.14728/) (2023)
- [Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change](https://mlsanity.com/paper/2210.17127/) (2022)
- [How Far Does BERT Look At:Distance-based Clustering and Analysis of BERT's Attention](https://mlsanity.com/paper/2011.00943/) (2020)
- [Substitution-based Semantic Change Detection using Contextual Embeddings](https://mlsanity.com/paper/2309.02403/) (2023)
- [Survey in Characterizing Semantic Change](https://mlsanity.com/paper/2402.19088/) (2024)
- [How BERT Speaks Shakespearean English? Evaluating Historical Bias in Contextual Language Models](https://mlsanity.com/paper/2402.05034/) (2024)

## Built on by (most cited first, 7 of 7)

- [Pretraining Language Models for Diachronic Linguistic Change Discovery](https://mlsanity.com/paper/2504.05523/) (2025)
- [Semantic Change Characterization with LLMs using Rhetorics](https://mlsanity.com/paper/2407.16624/) (2024)
- [SCDTour: Embedding Axis Ordering and Merging for Interpretable Semantic Change Detection](https://mlsanity.com/paper/2509.11818/) (2025)
- [Analyzing Continuous Semantic Shifts with Diachronic Word Similarity Matrices](https://mlsanity.com/paper/2501.09538/) (2025)
- [DHPLT: large-scale multilingual diachronic corpora and word representations for semantic change modelling](https://mlsanity.com/paper/2602.11968/) (2026)
- [Relation Geometry in Semantic Space of Language Models](https://mlsanity.com/paper/2607.26762/) (2026)
- ["I'm happy even though it's not real": GenAI Photo Editing as a Remembering Experience](https://mlsanity.com/paper/2602.03104/) (2026)

## Abstract (arXiv)

Modern language models are capable of contextualizing words based on their surrounding context. However, this capability is often compromised due to semantic change that leads to words being used in new, unexpected contexts not encountered during pre-training. In this paper, we model textitsemantic change by studying the effect of unexpected contexts introduced by textitlexical replacements. We propose a textitreplacement schema where a target word is substituted with lexical replacements of varying relatedness, thus simulating different kinds of semantic change. Furthermore, we leverage the replacement schema as a basis for a novel textitinterpretable model for semantic change. We are also the first to evaluate the use of LLaMa for semantic change detection.