mlsanityData to 7 Oct 2026

(Chat)GPT v BERT: Dawn of Justice for Semantic Change Detection

arXiv 2401.14040 ↗First posted 25 Jan 2024cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

In the universe of Natural Language Processing, Transformer-based language models like BERT and (Chat)GPT have emerged as lexical superheroes with great power to solve open research problems. In this paper, we specifically focus on the temporal problem of semantic change, and evaluate their ability to solve two diachronic extensions of the Word-in-Context (WiC) task: TempoWiC and HistoWiC. In particular, we investigate the potential of a novel, off-the-shelf technology like ChatGPT (and GPT) 3.5 compared to BERT, which represents a family of models that currently stand as the state-of-the-art for modeling semantic change. Our experiments represent the first attempt to assess the use of (Chat)GPT for studying semantic change. Our results indicate that ChatGPT performs significantly worse than the foundational GPT version. Furthermore, our results demonstrate that (Chat)GPT achieves slightly lower performance than BERT in detecting long-term changes but performs significantly worse in detecting short-term changes.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 27 of its 49 references that are in the map. First posted on arXiv 25 Jan 2024.

Builds on

27 of 49 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  5. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  6. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  7. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  8. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  9. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
  10. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  11. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  12. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
15 more
  1. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  2. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  3. 2020SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detectioncited by 105
  4. 2018Diachronic word embeddings and semantic shifts: a surveycited by 90
  5. 2018Survey of Computational Approaches to Lexical Semantic Changecited by 60
  6. 2023Will ChatGPT get you caught? Rethinking of Plagiarism Detectioncited by 41
  7. 2018Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Changecited by 35
  8. 2021Explaining and Improving BERT Performance on Lexical Semantic Change Detectioncited by 33
  9. 2019Universal Text Representation from BERT: An Empirical Studycited by 28
  10. 2021DWUG: A large Resource of Diachronic Word Usage Graphs in Four Languagescited by 25
  11. 2022TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Mediacited by 19
  12. 2018A State-of-the-Art of Semantic Change Computationcited by 14
  13. 2023Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligencecited by 13
  14. 2023The DURel Annotation Tool: Human and Computational Measurement of Semantic Proximity, Sense Clusters and Semantic Changecited by 7
  15. 2022Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiCcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 2.

7 papers.

  1. 2024A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Changecited by 21
  2. 2024Towards Automating Text Annotation: A Case Study on Semantic Proximity Annotation using GPT-4cited by 6
  3. 2024A Semantic Distance Metric Learning approach for Lexical Semantic Change Detectioncited by 5
  4. 2024AXOLOTL'24 Shared Task on Multilingual Explainable Semantic Change Modelingcited by 3
  5. 2024Tracing the Development of the Virtual Particle Concept Using Semantic Change Detectioncited by 3
  6. 2025Tougher Text, Smarter Models: Raising the Bar for Adversarial Defence Benchmarkscited by 2
  7. 2025Large Language Models for History, Philosophy, and Sociology of Science: Interpretive Uses, Methodological Challenges, and Critical Perspectivescited by 2