# (Chat)GPT v BERT: Dawn of Justice for Semantic Change Detection

arXiv: https://arxiv.org/abs/2401.14040 · HTML: https://mlsanity.com/paper/2401.14040/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 49 references that are in the map.
- First posted on arXiv 25 Jan 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT](https://mlsanity.com/paper/2302.09419/) (2023)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT](https://mlsanity.com/paper/2302.10198/) (2023)
- [Towards Making the Most of ChatGPT for Machine Translation](https://mlsanity.com/paper/2303.13780/) (2023)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)
- [SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection](https://mlsanity.com/paper/2007.11464/) (2020)
- [Diachronic word embeddings and semantic shifts: a survey](https://mlsanity.com/paper/1806.03537/) (2018)
- [Survey of Computational Approaches to Lexical Semantic Change](https://mlsanity.com/paper/1811.06278/) (2018)
- [Will ChatGPT get you caught? Rethinking of Plagiarism Detection](https://mlsanity.com/paper/2302.04335/) (2023)
- [Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change](https://mlsanity.com/paper/1804.06517/) (2018)
- [Explaining and Improving BERT Performance on Lexical Semantic Change Detection](https://mlsanity.com/paper/2103.07259/) (2021)
- [Universal Text Representation from BERT: An Empirical Study](https://mlsanity.com/paper/1910.07973/) (2019)
- [DWUG: A large Resource of Diachronic Word Usage Graphs in Four Languages](https://mlsanity.com/paper/2104.08540/) (2021)
- [TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media](https://mlsanity.com/paper/2209.07216/) (2022)
- [Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligence](https://mlsanity.com/paper/2309.14379/) (2023)
- [A State-of-the-Art of Semantic Change Computation](https://mlsanity.com/paper/1801.09872/) (2018)
- [The DURel Annotation Tool: Human and Computational Measurement of Semantic Proximity, Sense Clusters and Semantic Change](https://mlsanity.com/paper/2311.12664/) (2023)
- [Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiC](https://mlsanity.com/paper/2211.03466/) (2022)

## Built on by (most cited first, 8 of 8)

- [A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change](https://mlsanity.com/paper/2402.12011/) (2024)
- [Towards Automating Text Annotation: A Case Study on Semantic Proximity Annotation using GPT-4](https://mlsanity.com/paper/2407.04130/) (2024)
- [A Semantic Distance Metric Learning approach for Lexical Semantic Change Detection](https://mlsanity.com/paper/2403.00226/) (2024)
- [AXOLOTL'24 Shared Task on Multilingual Explainable Semantic Change Modeling](https://mlsanity.com/paper/2407.04079/) (2024)
- [Tracing the Development of the Virtual Particle Concept Using Semantic Change Detection](https://mlsanity.com/paper/2410.16855/) (2024)
- [Tougher Text, Smarter Models: Raising the Bar for Adversarial Defence Benchmarks](https://mlsanity.com/paper/2501.02654/) (2025)
- [Large Language Models for History, Philosophy, and Sociology of Science: Interpretive Uses, Methodological Challenges, and Critical Perspectives](https://mlsanity.com/paper/2506.12242/) (2025)
- [The BD-LSC Dataset: Facilitating the Benchmarking of Models for Lexical Semantic Change Detection in Slang and Standard Usage](https://mlsanity.com/paper/2606.16560/) (2026)

## Abstract (arXiv)

In the universe of Natural Language Processing, Transformer-based language models like BERT and (Chat)GPT have emerged as lexical superheroes with great power to solve open research problems. In this paper, we specifically focus on the temporal problem of semantic change, and evaluate their ability to solve two diachronic extensions of the Word-in-Context (WiC) task: TempoWiC and HistoWiC. In particular, we investigate the potential of a novel, off-the-shelf technology like ChatGPT (and GPT) 3.5 compared to BERT, which represents a family of models that currently stand as the state-of-the-art for modeling semantic change. Our experiments represent the first attempt to assess the use of (Chat)GPT for studying semantic change. Our results indicate that ChatGPT performs significantly worse than the foundational GPT version. Furthermore, our results demonstrate that (Chat)GPT achieves slightly lower performance than BERT in detecting long-term changes but performs significantly worse in detecting short-term changes.