# Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similarity

arXiv: https://arxiv.org/abs/2003.04866 · HTML: https://mlsanity.com/paper/2003.04866/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Mar 2020.

## Built on by (most cited first, 30 of 38)

- [Visually Grounded Reasoning across Languages and Cultures](https://mlsanity.com/paper/2109.13238/) (2021)
- [Sustainable Modular Debiasing of Language Models](https://mlsanity.com/paper/2109.03646/) (2021)
- [Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders](https://mlsanity.com/paper/2104.08027/) (2021)
- [Multilingual Language Models Predict Human Reading Behavior](https://mlsanity.com/paper/2104.05433/) (2021)
- [On the Independence of Association Bias and Empirical Fairness in Language Models](https://mlsanity.com/paper/2304.10153/) (2023)
- [Revisiting non-English Text Simplification: A Unified Multilingual Benchmark](https://mlsanity.com/paper/2305.15678/) (2023)
- [On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs](https://mlsanity.com/paper/2407.19200/) (2024)
- [Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders](https://mlsanity.com/paper/2104.08757/) (2021)
- [Measuring Fairness with Biased Rulers: A Survey on Quantifying Biases in Pretrained Language Models](https://mlsanity.com/paper/2112.07447/) (2021)
- [Improving Word Translation via Two-Stage Contrastive Learning](https://mlsanity.com/paper/2203.08307/) (2022)
- [SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages](https://mlsanity.com/paper/2402.08638/) (2024)
- [AM2iCo: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examples](https://mlsanity.com/paper/2104.08639/) (2021)
- [Visual Comparison of Language Model Adaptation](https://mlsanity.com/paper/2208.08176/) (2022)
- [VAST: The Valence-Assessing Semantics Test for Contextualizing Language Models](https://mlsanity.com/paper/2203.07504/) (2022)
- [SimRelUz: Similarity and Relatedness scores as a Semantic Evaluation dataset for Uzbek language](https://mlsanity.com/paper/2205.06072/) (2022)
- [Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders](https://mlsanity.com/paper/2205.00267/) (2022)
- [Improving Bilingual Lexicon Induction with Cross-Encoder Reranking](https://mlsanity.com/paper/2210.16953/) (2022)
- [GrEmLIn: A Repository of Green Baseline Embeddings for 87 Low-Resource Languages Injected with Multilingual Graph Knowledge](https://mlsanity.com/paper/2409.18193/) (2024)
- [Quantifying syntax similarity with a polynomial representation of dependency trees](https://mlsanity.com/paper/2211.07005/) (2022)
- [Bridging Fairness and Environmental Sustainability in Natural Language Processing](https://mlsanity.com/paper/2211.04256/) (2022)
- [Beyond Film Subtitles: Is YouTube the Best Approximation of Spoken Vocabulary?](https://mlsanity.com/paper/2410.03240/) (2024)
- [When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Training](https://mlsanity.com/paper/2601.22851/) (2026)
- [Towards Debiasing Translation Artifacts](https://mlsanity.com/paper/2205.08001/) (2022)
- [Back to the Future: On Potential Histories in NLP](https://mlsanity.com/paper/2210.06245/) (2022)
- [What do Large Language Models Learn beyond Language?](https://mlsanity.com/paper/2210.12302/) (2022)
- [The Causal Influence of Grammatical Gender on Distributional Semantics](https://mlsanity.com/paper/2311.18567/) (2023)
- [What is "Typological Diversity" in NLP?](https://mlsanity.com/paper/2402.04222/) (2024)
- [Local Contrastive Editing of Gender Stereotypes](https://mlsanity.com/paper/2410.17739/) (2024)
- [Multilingual CheckList: Generation and Evaluation](https://mlsanity.com/paper/2203.12865/) (2022)
- [SuperSim: a test set for word similarity and relatedness in Swedish](https://mlsanity.com/paper/2104.05228/) (2021)

## Abstract (arXiv)

We introduce Multi-SimLex, a large-scale lexical resource and evaluation benchmark covering datasets for 12 typologically diverse languages, including major languages (e.g., Mandarin Chinese, Spanish, Russian) as well as less-resourced ones (e.g., Welsh, Kiswahili). Each language dataset is annotated for the lexical relation of semantic similarity and contains 1,888 semantically aligned concept pairs, providing a representative coverage of word classes (nouns, verbs, adjectives, adverbs), frequency ranks, similarity intervals, lexical fields, and concreteness levels. Additionally, owing to the alignment of concepts across languages, we provide a suite of 66 cross-lingual semantic similarity datasets. Due to its extensive size and language coverage, Multi-SimLex provides entirely novel opportunities for experimental evaluation and analysis. On its monolingual and cross-lingual benchmarks, we evaluate and analyze a wide array of recent state-of-the-art monolingual and cross-lingual representation models, including static and contextualized word embeddings (such as fastText, M-BERT and XLM), externally informed lexical representations, as well as fully unsupervised and (weakly) supervised cross-lingual word embeddings. We also present a step-by-step dataset creation protocol for creating consistent, Multi-Simlex-style resources for additional languages. We make these contributions -- the public release of Multi-SimLex datasets, their creation protocol, strong baseline results, and in-depth analyses which can be be helpful in guiding future developments in multilingual lexical semantics and representation learning -- available via a website which will encourage community effort in further expansion of Multi-Simlex to many more languages. Such a large-scale semantic resource could inspire significant further advances in NLP across languages.