# AM2iCo: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examples

arXiv: https://arxiv.org/abs/2104.08639 · HTML: https://mlsanity.com/paper/2104.08639/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 17 Apr 2021.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing](https://mlsanity.com/paper/1807.00914/) (2018)
- [On the Limitations of Unsupervised Bilingual Dictionary Induction](https://mlsanity.com/paper/1805.03620/) (2018)
- [How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions](https://mlsanity.com/paper/1902.00508/) (2019)
- [XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization](https://mlsanity.com/paper/2010.06478/) (2020)
- [DeepType: Multilingual Entity Linking by Neural Type System Evolution](https://mlsanity.com/paper/1802.01021/) (2018)
- [Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similarity](https://mlsanity.com/paper/2003.04866/) (2020)
- [Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages](https://mlsanity.com/paper/2001.11453/) (2020)
- [Joint Multilingual Supervision for Cross-lingual Entity Linking](https://mlsanity.com/paper/1809.07657/) (2018)
- [CLUSE: Cross-Lingual Unsupervised Sense Embeddings](https://mlsanity.com/paper/1809.05694/) (2018)

## Built on by (most cited first, 14 of 14)

- [NusaCrowd: Open Source Initiative for Indonesian NLP Resources](https://mlsanity.com/paper/2212.09648/) (2022)
- [TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media](https://mlsanity.com/paper/2209.07216/) (2022)
- [MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models](https://mlsanity.com/paper/2109.09237/) (2021)
- [Recent Advancements and Challenges of Turkic Central Asian Language Processing](https://mlsanity.com/paper/2407.05006/) (2024)
- [A Semantic Distance Metric Learning approach for Lexical Semantic Change Detection](https://mlsanity.com/paper/2403.00226/) (2024)
- [Investigating the Contextualised Word Embedding Dimensions Specified for Contextual and Temporal Semantic Changes](https://mlsanity.com/paper/2407.02820/) (2024)
- [SenWiCh: Sense-Annotation of Low-Resource Languages for WiC using Hybrid Methods](https://mlsanity.com/paper/2505.23714/) (2025)
- [Measuring Context-Word Biases in Lexical Semantic Datasets](https://mlsanity.com/paper/2112.06733/) (2021)
- [What Drives Cross-lingual Ranking? Retrieval Approaches with Multilingual Language Models](https://mlsanity.com/paper/2511.19324/) (2025)
- [Improving Interpretability of Lexical Semantic Change with Neurobiological Features](https://mlsanity.com/paper/2602.09760/) (2026)
- [Distilling Monolingual and Crosslingual Word-in-Context Representations](https://mlsanity.com/paper/2409.08719/) (2024)
- [Multilinguality Does not Make Sense: Investigating Factors Behind Zero-Shot Transfer in Sense-Aware Tasks](https://mlsanity.com/paper/2505.24834/) (2025)
- [Automatically Suggesting Diverse Example Sentences for L2 Japanese Learners Using Pre-Trained Language Models](https://mlsanity.com/paper/2506.03580/) (2025)
- [XL-DURel: Finetuning Sentence Transformers for Ordinal Word-in-Context Classification](https://mlsanity.com/paper/2507.14578/) (2025)

## Abstract (arXiv)

Capturing word meaning in context and distinguishing between correspondences and variations across languages is key to building successful multilingual and cross-lingual text representation models. However, existing multilingual evaluation datasets that evaluate lexical semantics "in-context" have various limitations. In particular, 1) their language coverage is restricted to high-resource languages and skewed in favor of only a few language families and areas, 2) a design that makes the task solvable via superficial cues, which results in artificially inflated (and sometimes super-human) performances of pretrained encoders, on many target languages, which limits their usefulness for model probing and diagnostics, and 3) little support for cross-lingual evaluation. In order to address these gaps, we present AM2iCo (Adversarial and Multilingual Meaning in Context), a wide-coverage cross-lingual and multilingual evaluation set; it aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts for 14 language pairs. We conduct a series of experiments in a wide range of setups and demonstrate the challenging nature of AM2iCo. The results reveal that current SotA pretrained encoders substantially lag behind human performance, and the largest gaps are observed for low-resource languages and languages dissimilar to English.