# Evaluating the Evaluator: Problems with SemEval-2020 Task 1 for Lexical Semantic Change Detection

arXiv: https://arxiv.org/abs/2604.13232 · HTML: https://mlsanity.com/paper/2604.13232/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 40 references that are in the map.
- First posted on arXiv 14 Apr 2026.

## Builds on (most cited first, 19 of 19)

- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016)
- [Large Language Models for Data Annotation and Synthesis: A Survey](https://mlsanity.com/paper/2402.13446/) (2024)
- [SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection](https://mlsanity.com/paper/2007.11464/) (2020)
- [Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change](https://mlsanity.com/paper/1804.06517/) (2018)
- [Explaining and Improving BERT Performance on Lexical Semantic Change Detection](https://mlsanity.com/paper/2103.07259/) (2021)
- [Contextualized language models for semantic change detection: lessons learned](https://mlsanity.com/paper/2209.00154/) (2022)
- [TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media](https://mlsanity.com/paper/2209.07216/) (2022)
- [LSCDiscovery: A shared task on semantic change discovery and detection in Spanish](https://mlsanity.com/paper/2205.06691/) (2022)
- [Grammatical Profiling for Semantic Change Detection](https://mlsanity.com/paper/2109.10397/) (2021)
- [Graph-based Clustering for Detecting Semantic Change Across Time and Languages](https://mlsanity.com/paper/2402.01025/) (2024)
- [Using Synchronic Definitions and Semantic Relations to Classify Semantic Change Types](https://mlsanity.com/paper/2406.03452/) (2024)
- [NorDiaChange: Diachronic Semantic Change Dataset for Norwegian](https://mlsanity.com/paper/2201.05123/) (2022)
- [AXOLOTL'24 Shared Task on Multilingual Explainable Semantic Change Modeling](https://mlsanity.com/paper/2407.04079/) (2024)
- [Transparent Semantic Change Detection with Dependency-Based Profiles](https://mlsanity.com/paper/2601.02891/) (2026)
- [ReFRAME or Remain: Unsupervised Lexical Semantic Change Detection with Frame Semantics](https://mlsanity.com/paper/2602.04514/) (2026)
- [Improving Interpretability of Lexical Semantic Change with Neurobiological Features](https://mlsanity.com/paper/2602.09760/) (2026)

## Built on by (most cited first, 1 of 1)

- [The BD-LSC Dataset: Facilitating the Benchmarking of Models for Lexical Semantic Change Detection in Slang and Standard Usage](https://mlsanity.com/paper/2606.16560/) (2026)

## Abstract (arXiv)

This discussion paper re-examines SemEval-2020 Task 1, the most influential shared benchmark for lexical semantic change detection, through a three-part evaluative framework: operationalisation, data quality, and benchmark design. First, at the level of operationalisation, we argue that the benchmark models semantic change mainly as gain, loss, or redistribution of discrete senses. While practical for annotation and evaluation, this framing is too narrow to capture gradual, constructional, collocational, and discourse-level change. Also, the gold labels are outcomes of annotation decisions, clustering procedures, and threshold settings, which could potentially limit the validity of the task. Second, at the level of data quality, we show that the benchmark is affected by substantial corpus and preprocessing problems, including OCR noise, malformed characters, truncated sentences, inconsistent lemmatisation, POS-tagging errors, and missed targets. These issues can distort model behaviour, complicate linguistic analysis, and reduce reproducibility. Third, at the level of bench-mark design, we argue the small curated target sets and limited language coverage reduce realism and increase statistical uncertainty. Taken together, these limitations suggest that the benchmark should be treated as a useful but partial test bed rather than a definitive measure of progress. We therefore call for future datasets and shared tasks to adopt broader theories of semantic change, document pre-processing transparently, expand cross-linguistic coverage, and use more realistic evaluation settings. Such steps are necessary for more valid, interpretable, and generalisable progress in lexical semantic change detection