# XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization

arXiv: https://arxiv.org/abs/2010.06478 · HTML: https://mlsanity.com/paper/2010.06478/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2020.

## Built on by (most cited first, 30 of 42)

- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages](https://mlsanity.com/paper/2501.06346/) (2025)
- [P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs](https://mlsanity.com/paper/2411.09116/) (2024)
- [Revisiting non-English Text Simplification: A Unified Multilingual Benchmark](https://mlsanity.com/paper/2305.15678/) (2023)
- [A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change](https://mlsanity.com/paper/2402.12011/) (2024)
- [TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media](https://mlsanity.com/paper/2209.07216/) (2022)
- [MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models](https://mlsanity.com/paper/2109.09237/) (2021)
- [Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words](https://mlsanity.com/paper/2501.06254/) (2025)
- [AM2iCo: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examples](https://mlsanity.com/paper/2104.08639/) (2021)
- [Prompting with Pseudo-Code Instructions](https://mlsanity.com/paper/2305.11790/) (2023)
- [Realistic Evaluation of Model Merging for Compositional Generalization](https://mlsanity.com/paper/2409.18314/) (2024)
- [Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy](https://mlsanity.com/paper/2106.13553/) (2021)
- [EntityCS: Improving Zero-Shot Cross-lingual Transfer with Entity-Centric Code Switching](https://mlsanity.com/paper/2210.12540/) (2022)
- [DUMB: A Benchmark for Smart Evaluation of Dutch Models](https://mlsanity.com/paper/2305.13026/) (2023)
- [Multilingual Prompt Engineering in Large Language Models: A Survey Across NLP Tasks](https://mlsanity.com/paper/2505.11665/) (2025)
- [GeoGLUE: A GeoGraphic Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2305.06545/) (2023)
- [A Semantic Distance Metric Learning approach for Lexical Semantic Change Detection](https://mlsanity.com/paper/2403.00226/) (2024)
- [MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch](https://mlsanity.com/paper/2509.12340/) (2025)
- [Investigating the Contextualised Word Embedding Dimensions Specified for Contextual and Temporal Semantic Changes](https://mlsanity.com/paper/2407.02820/) (2024)
- [Thank You, Stingray: Multilingual Large Language Models Can Not (Yet) Disambiguate Cross-Lingual Word Sense](https://mlsanity.com/paper/2410.21573/) (2024)
- [SenWiCh: Sense-Annotation of Low-Resource Languages for WiC using Hybrid Methods](https://mlsanity.com/paper/2505.23714/) (2025)
- [WiC = TSV = WSD: On the Equivalence of Three Semantic Tasks](https://mlsanity.com/paper/2107.14352/) (2021)
- [Blessing of Multilinguality: A Systematic Analysis of Multilingual In-Context Learning](https://mlsanity.com/paper/2502.11364/) (2025)
- [Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiC](https://mlsanity.com/paper/2211.03466/) (2022)
- [Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Models](https://mlsanity.com/paper/2304.13803/) (2023)
- [Empowering Persian LLMs for Instruction Following: A Novel Dataset and Training Approach](https://mlsanity.com/paper/2407.11186/) (2024)
- [MCL@IITK at SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation using Augmented Data, Signals, and Transformers](https://mlsanity.com/paper/2104.01567/) (2021)
- [EENLP: Cross-lingual Eastern European NLP Index](https://mlsanity.com/paper/2108.02605/) (2021)
- [Measuring Context-Word Biases in Lexical Semantic Datasets](https://mlsanity.com/paper/2112.06733/) (2021)

## Abstract (arXiv)

The ability to correctly model distinct meanings of a word is crucial for the effectiveness of semantic representation techniques. However, most existing evaluation benchmarks for assessing this criterion are tied to sense inventories (usually WordNet), restricting their usage to a small subset of knowledge-based representation techniques. The Word-in-Context dataset (WiC) addresses the dependence on sense inventories by reformulating the standard disambiguation task as a binary classification problem; but, it is limited to the English language. We put forward a large multilingual benchmark, XL-WiC, featuring gold standards in 12 new languages from varied language families and with different degrees of resource availability, opening room for evaluation scenarios such as zero-shot cross-lingual transfer. We perform a series of experiments to determine the reliability of the datasets and to set performance baselines for several recent contextualized multilingual models. Experimental results show that even when no tagged instances are available for a target language, models trained solely on the English data can attain competitive performance in the task of distinguishing different meanings of a word, even for distant languages. XL-WiC is available at https://pilehvar.github.io/xlwic/.