# Multilingual LLMs Struggle to Link Orthography and Semantics in Bilingual Word Processing

arXiv: https://arxiv.org/abs/2501.09127 · HTML: https://mlsanity.com/paper/2501.09127/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 78 references that are in the map.
- First posted on arXiv 15 Jan 2025.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Evaluating Large Language Models in Theory of Mind Tasks](https://mlsanity.com/paper/2302.02083/) (2023)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024)
- [What Large Language Models Know and What People Think They Know](https://mlsanity.com/paper/2401.13835/) (2024)
- [Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMs](https://mlsanity.com/paper/2305.16339/) (2023)
- [Language Models are Few-shot Multilingual Learners](https://mlsanity.com/paper/2109.07684/) (2021)
- [Improving Multilingual Models with Language-Clustered Vocabularies](https://mlsanity.com/paper/2010.12777/) (2020)
- [Language Model Alignment in Multilingual Trolley Problems](https://mlsanity.com/paper/2407.02273/) (2024)
- [When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer](https://mlsanity.com/paper/2110.14782/) (2021)
- [Probing the Emergence of Cross-lingual Alignment during LLM Training](https://mlsanity.com/paper/2406.13229/) (2024)
- [Do large language models resemble humans in language use?](https://mlsanity.com/paper/2303.08014/) (2023)
- [Contextual Feature Extraction Hierarchies Converge in Large Language Models and the Brain](https://mlsanity.com/paper/2401.17671/) (2024)
- [Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training](https://mlsanity.com/paper/2109.07306/) (2021)
- [Large Language Models Lack Understanding of Character Composition of Words](https://mlsanity.com/paper/2405.11357/) (2024)
- [mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?](https://mlsanity.com/paper/2404.12444/) (2024)
- [Learning an Executable Neural Semantic Parser](https://mlsanity.com/paper/1711.05066/) (2017)
- [What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structure](https://mlsanity.com/paper/2302.12239/) (2023)

## Built on by (most cited first, 3 of 3)

- [Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models](https://mlsanity.com/paper/2511.21086/) (2025)
- [When Similar Means Different: Evaluating LLMs on Arabic--Hebrew Cognates](https://mlsanity.com/paper/2606.13218/) (2026)
- [Tokenizing Crosslingual Homographs](https://mlsanity.com/paper/2607.17689/) (2026)

## Abstract (arXiv)

Bilingual lexical processing is shaped by the complex interplay of phonological, orthographic, and semantic features of two languages within an integrated mental lexicon. In humans, this is evident in the ease with which cognate words - words similar in both orthographic form and meaning (e.g., blind, meaning "sightless" in both English and German) - are processed, compared to the challenges posed by interlingual homographs, which share orthographic form but differ in meaning (e.g., gift, meaning "present" in English but "poison" in German). We investigate how multilingual Large Language Models (LLMs) handle such phenomena, focusing on English-Spanish, English-French, and English-German cognates, non-cognate, and interlingual homographs. Specifically, we evaluate their ability to disambiguate meanings and make semantic judgments, both when these word types are presented in isolation or within sentence contexts. Our findings reveal that while certain LLMs demonstrate strong performance in recognizing cognates and non-cognates in isolation, they exhibit significant difficulty in disambiguating interlingual homographs, often performing below random baselines. This suggests LLMs tend to rely heavily on orthographic similarities rather than semantic understanding when interpreting interlingual homographs. Further, we find LLMs exhibit difficulty in retrieving word meanings, with performance in isolative disambiguation tasks having no correlation with semantic understanding. Finally, we study how the LLM processes interlingual homographs in incongruent sentences. We find models to opt for different strategies in understanding English and non-English homographs, highlighting a lack of a unified approach to handling cross-lingual ambiguities.