mlsanityData to 7 Oct 2026

Thank You, Stingray: Multilingual Large Language Models Can Not (Yet) Disambiguate Cross-Lingual Word Sense

arXiv 2410.21573 ↗First posted 28 Oct 2024cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Multilingual large language models (LLMs) have gained prominence, but concerns arise regarding their reliability beyond English. This study addresses the gap in cross-lingual semantic evaluation by introducing a novel benchmark for cross-lingual sense disambiguation, StingrayBench. In this paper, we demonstrate using false friends -- words that are orthographically similar but have completely different meanings in two languages -- as a possible approach to pinpoint the limitation of cross-lingual sense disambiguation in LLMs. We collect false friends in four language pairs, namely Indonesian-Malay, Indonesian-Tagalog, Chinese-Japanese, and English-German; and challenge LLMs to distinguish the use of them in context. In our analysis of various models, we observe they tend to be biased toward higher-resource languages. We also propose new metrics for quantifying the cross-lingual sense bias and comprehension based on our benchmark. Our work contributes to developing more diverse and inclusive language modeling, promoting fairer access for the wider multilingual community.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 31 of its 69 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

31 of 69 references.

References · 31

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  8. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  9. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  10. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  11. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  12. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
19 more
  1. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  2. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  3. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  4. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  5. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 67
  6. 2023Multilingual Large Language Models Are Not (Yet) Code-Switcherscited by 65
  7. 2020Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoderscited by 61
  8. 2021IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generationcited by 57
  9. 2023Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languagescited by 50
  10. 2023Multi-lingual and Multi-cultural Figurative Language Understandingcited by 47
  11. 2020XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualizationcited by 42
  12. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 27
  13. 2024On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Taskcited by 24
  14. 2023InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuningcited by 21
  15. 2024Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languagescited by 18
  16. 2023CroCoSum: A Benchmark Dataset for Cross-Lingual Code-Switched Summarizationcited by 8
  17. 2024Automated Cognate Detection as a Supervised Link Prediction Task with Cognate Transformercited by 6
  18. 2022Multilingual Word Sense Disambiguation with Unified Sense Representationcited by 5
  19. 2023Combating the Curse of Multilinguality in Cross-Lingual WSD by Aligning Sparse Contextualized Word Representationscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025FilBench: Can LLMs Understand and Generate Filipino?cited by 3
  2. 2025Prompt Balance Matters: Understanding How Imbalanced Few-Shot Learning Affects Multilingual Sense Disambiguation in LLMscited by 1
  3. 2026An Exploration-Analysis-Disambiguation Reasoning Framework for Word Sense Disambiguation with Low-Parameter LLMscited by 1
  4. 2026SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguationcited by 0