# textitSwap and Predict -- Predicting the Semantic Changes in Words across Corpora by Context Swapping

arXiv: https://arxiv.org/abs/2310.10397 · HTML: https://mlsanity.com/paper/2310.10397/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 16 Oct 2023.

## Builds on (most cited first, 24 of 24)

- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [Mind the Gap: Assessing Temporal Generalization in Neural Language Models](https://mlsanity.com/paper/2102.01951/) (2021)
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016)
- [TimeLMs: Diachronic Language Models from Twitter](https://mlsanity.com/paper/2202.03829/) (2022)
- [SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection](https://mlsanity.com/paper/2007.11464/) (2020)
- [Evaluating the Underlying Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.08783/) (2019)
- [Diachronic word embeddings and semantic shifts: a survey](https://mlsanity.com/paper/1806.03537/) (2018)
- [Analysing Lexical Semantic Change with Contextualised Word Representations](https://mlsanity.com/paper/2004.14118/) (2020)
- [Statistically Significant Detection of Linguistic Change](https://mlsanity.com/paper/1411.3315/) (2014)
- [Survey of Computational Approaches to Lexical Semantic Change](https://mlsanity.com/paper/1811.06278/) (2018)
- [Temporal Analysis of Language through Neural Language Models](https://mlsanity.com/paper/1405.3515/) (2014)
- [Time Masking for Temporal Language Models](https://mlsanity.com/paper/2110.06366/) (2021)
- [Dynamic Word Embeddings for Evolving Semantic Discovery](https://mlsanity.com/paper/1703.00607/) (2017)
- [Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shift](https://mlsanity.com/paper/1912.01072/) (2019)
- [Explaining and Improving BERT Performance on Lexical Semantic Change Detection](https://mlsanity.com/paper/2103.07259/) (2021)
- [Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change](https://mlsanity.com/paper/1906.01688/) (2019)
- [UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection](https://mlsanity.com/paper/2005.00050/) (2020)
- [Short-Term Meaning Shift: A Distributional Exploration](https://mlsanity.com/paper/1809.03169/) (2018)
- [Temporal Attention for Language Models](https://mlsanity.com/paper/2202.02093/) (2022)
- [Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change](https://mlsanity.com/paper/2210.17127/) (2022)
- [Statistically significant detection of semantic shifts using contextual word embeddings](https://mlsanity.com/paper/2104.03776/) (2021)
- [Unsupervised Semantic Variation Prediction using the Distribution of Sibling Embeddings](https://mlsanity.com/paper/2305.08654/) (2023)
- [Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy](https://mlsanity.com/paper/2110.02204/) (2021)

## Built on by (most cited first, 6 of 6)

- [A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change](https://mlsanity.com/paper/2402.12011/) (2024)
- [A Semantic Distance Metric Learning approach for Lexical Semantic Change Detection](https://mlsanity.com/paper/2403.00226/) (2024)
- [Investigating the Contextualised Word Embedding Dimensions Specified for Contextual and Temporal Semantic Changes](https://mlsanity.com/paper/2407.02820/) (2024)
- [SCDTour: Embedding Axis Ordering and Merging for Interpretable Semantic Change Detection](https://mlsanity.com/paper/2509.11818/) (2025)
- [Improving Interpretability of Lexical Semantic Change with Neurobiological Features](https://mlsanity.com/paper/2602.09760/) (2026)
- [HistLens: Mapping Idea Change across Concepts and Corpora](https://mlsanity.com/paper/2604.11749/) (2026)

## Abstract (arXiv)

Meanings of words change over time and across domains. Detecting the semantic changes of words is an important task for various NLP applications that must make time-sensitive predictions. We consider the problem of predicting whether a given target word, w, changes its meaning between two different text corpora, C1 and C2. For this purpose, we propose textitSwapping-based Semantic Change Detection (SSCD), an unsupervised method that randomly swaps contexts between C1 and C2 where w occurs. We then look at the distribution of contextualised word embeddings of w, obtained from a pretrained masked language model (MLM), representing the meaning of w in its occurrence contexts in C1 and C2. Intuitively, if the meaning of w does not change between C1 and C2, we would expect the distributions of contextualised word embeddings of w to remain the same before and after this random swapping process. Despite its simplicity, we demonstrate that even by using pretrained MLMs without any fine-tuning, our proposed context swapping method accurately predicts the semantic changes of words in four languages (English, German, Swedish, and Latin) and across different time spans (over 50 years and about five years). Moreover, our method achieves significant performance improvements compared to strong baselines for the English semantic change prediction task. Source code is available at https://github.com/a1da4/svp-swap .