# LSC-Eval: A General Framework to Evaluate Methods for Assessing Dimensions of Lexical Semantic Change Using LLM-Generated Synthetic Data

arXiv: https://arxiv.org/abs/2503.08042 · HTML: https://mlsanity.com/paper/2503.08042/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 68 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection](https://mlsanity.com/paper/2007.11464/) (2020)
- [Best Practices and Lessons Learned on Synthetic Data](https://mlsanity.com/paper/2404.07503/) (2024)
- [Diachronic word embeddings and semantic shifts: a survey](https://mlsanity.com/paper/1806.03537/) (2018)
- [Analysing Lexical Semantic Change with Contextualised Word Representations](https://mlsanity.com/paper/2004.14118/) (2020)
- [LLMs Are Few-Shot In-Context Low-Resource Language Learners](https://mlsanity.com/paper/2403.16512/) (2024)
- [Multilingual Language Models are not Multicultural: A Case Study in Emotion](https://mlsanity.com/paper/2307.01370/) (2023)
- [How Many Demonstrations Do You Need for In-context Learning?](https://mlsanity.com/paper/2303.08119/) (2023)
- [Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change](https://mlsanity.com/paper/1906.01688/) (2019)
- [A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change](https://mlsanity.com/paper/2402.12011/) (2024)
- [TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media](https://mlsanity.com/paper/2209.07216/) (2022)
- [Challenges for Computational Lexical Semantic Change](https://mlsanity.com/paper/2101.07668/) (2021)
- [A State-of-the-Art of Semantic Change Computation](https://mlsanity.com/paper/1801.09872/) (2018)
- [Using Synchronic Definitions and Semantic Relations to Classify Semantic Change Types](https://mlsanity.com/paper/2406.03452/) (2024)
- [Survey in Characterizing Semantic Change](https://mlsanity.com/paper/2402.19088/) (2024)
- [Semantic Change Characterization with LLMs using Rhetorics](https://mlsanity.com/paper/2407.16624/) (2024)
- [It is Simple Sometimes: A Study On Improving Aspect-Based Sentiment Analysis Performance](https://mlsanity.com/paper/2405.20703/) (2024)
- [A Multidimensional Framework for Evaluating Lexical Semantic Change with Social Science Applications](https://mlsanity.com/paper/2406.06052/) (2024)
- [Generating bilingual example sentences with large language models as lexicography assistants](https://mlsanity.com/paper/2410.03182/) (2024)
- [Analyzing Continuous Semantic Shifts with Diachronic Word Similarity Matrices](https://mlsanity.com/paper/2501.09538/) (2025)

## Built on by (most cited first, 2 of 2)

- [Multilinguality Does not Make Sense: Investigating Factors Behind Zero-Shot Transfer in Sense-Aware Tasks](https://mlsanity.com/paper/2505.24834/) (2025)
- [Automatic Evaluation of Mental Health Stigma in Online Communication](https://mlsanity.com/paper/2610.02775/) (2026)

## Abstract (arXiv)

Lexical Semantic Change (LSC) provides insight into cultural and social dynamics. Yet, the validity of methods for measuring different kinds of LSC remains unestablished due to the absence of historical benchmark datasets. To address this gap, we propose LSC-Eval, a novel three-stage general-purpose evaluation framework to: (1) develop a scalable methodology for generating synthetic datasets that simulate theory-driven LSC using In-Context Learning and a lexical database; (2) use these datasets to evaluate the sensitivity of computational methods to synthetic change; and (3) assess their suitability for detecting change in specific dimensions and domains. We apply LSC-Eval to simulate changes along the Sentiment, Intensity, and Breadth (SIB) dimensions, as defined in the SIBling framework, using examples from psychology. We then evaluate the ability of selected methods to detect these controlled interventions. Our findings validate the use of synthetic benchmarks, demonstrate that tailored methods effectively detect changes along SIB dimensions, and reveal that a state-of-the-art LSC model faces challenges in detecting affective dimensions of LSC. LSC-Eval offers a valuable tool for dimension- and domain-specific benchmarking of LSC methods, with particular relevance to the social sciences.