SemBench: A Universal Semantic Framework for LLM Evaluation
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Recent progress in Natural Language Processing (NLP) has been driven by the emergence of Large Language Models (LLMs), which exhibit remarkable generative and reasoning capabilities. However, despite their success, evaluating the true semantic understanding of these models remains a persistent challenge. Traditional benchmarks such as Word-in-Context (WiC) effectively probe this capability, but their creation is resource-intensive and often limited to high-resource languages. In this paper, we introduce SemBench, a framework for automatically generating synthetic benchmarks that assess the semantic competence of LLMs using only dictionary sense definitions and a sentence encoder. This approach eliminates the need for curated example sentences, making it both scalable and language-independent. We evaluate SemBench in three languages (English, Spanish, and Basque) spanning different levels of linguistic resources, and across a wide range of LLMs. Our results show that rankings derived from SemBench strongly correlate with those obtained from standard WiC datasets. Furthermore, our analysis demonstrates that only a small number of examples is required to achieve stable and meaningful rankings. Overall, SemBench provides a lightweight, adaptable, and data-efficient framework for cross-lingual evaluation of semantic understanding in LLMs.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 17 of its 43 references that are in the map. First posted on arXiv 12 Mar 2026.
Builds on
17 of 43 references.
References · 17
- 2017Attention Is All You Needcited by 60,361
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
- 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
- 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 172
- 2025A Survey on Code Generation with LLM-based Agentscited by 106
- 2024LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptionscited by 47
- 2016Definition Modeling: Learning to define word embeddings in natural languagecited by 46
- 2018Conditional Generators of Words Definitionscited by 35
- 2023Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysiscited by 19
5 more
- 2024Towards Rationality in Language and Multimodal Agents: A Surveycited by 16
- 2025Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basquecited by 12
- 2022Fine-grained Contrastive Learning for Definition Generationcited by 10
- 2024Low-Cost Generation and Evaluation of Dictionary Example Sentencescited by 4
- 2025SenWiCh: Sense-Annotation of Low-Resource Languages for WiC using Hybrid Methodscited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Through the Lens of Core Competency: Survey on Evaluation of Large Language Models
- 2023NLPBench: Evaluating Large Language Models on Solving NLP Problems
- 2024A Systematic Evaluation of Large Language Models for Natural Language Generation Tasks
- 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models
- 2024Enhancing LLM Evaluations: The Garbling Trick
- 2025Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements
- 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
- 2025SELT: Self-Evaluation Tree Search for LLMs with Task Decomposition