# When Similar Means Different: Evaluating LLMs on Arabic--Hebrew Cognates

arXiv: https://arxiv.org/abs/2606.13218 · HTML: https://mlsanity.com/paper/2606.13218/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 63 references that are in the map.
- First posted on arXiv 11 Jun 2026.

## Builds on (most cited first, 19 of 19)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [From SPMRL to NMRL: What Did We Learn (and Unlearn) in a Decade of Parsing Morphologically-Rich Languages (MRLs)?](https://mlsanity.com/paper/2005.01330/) (2020)
- [Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary and Instruction Capabilities](https://mlsanity.com/paper/2407.07080/) (2024)
- [The FIGNEWS Shared Task on News Media Narratives](https://mlsanity.com/paper/2407.18147/) (2024)
- [Automated Cognate Detection as a Supervised Link Prediction Task with Cognate Transformer](https://mlsanity.com/paper/2402.02926/) (2024)
- [Splintering Nonconcatenative Languages for Better Tokenization](https://mlsanity.com/paper/2503.14433/) (2025)
- [Harnessing Cross-lingual Features to Improve Cognate Detection for Low-resource Languages](https://mlsanity.com/paper/2112.08789/) (2021)
- [Cognate Transformer for Automated Phonological Reconstruction and Cognate Reflex Prediction](https://mlsanity.com/paper/2310.07487/) (2023)
- [IMPACT: Inflectional Morphology Probes Across Complex Typologies](https://mlsanity.com/paper/2506.23929/) (2025)
- [Cognition-aware Cognate Detection](https://mlsanity.com/paper/2112.08087/) (2021)
- [Multilingual LLMs Struggle to Link Orthography and Semantics in Bilingual Word Processing](https://mlsanity.com/paper/2501.09127/) (2025)
- [An Arabic-Hebrew parallel corpus of TED talks](https://mlsanity.com/paper/1610.00572/) (2016)
- [Large-Scale Machine Translation between Arabic and Hebrew: Available Corpora and Initial Results](https://mlsanity.com/paper/1609.07701/) (2016)
- [A Tale of Two Scripts: Transliteration and Post-Correction for Judeo-Arabic](https://mlsanity.com/paper/2507.04746/) (2025)

## Abstract (arXiv)

Arabic and Hebrew, as closely related Semitic languages, share many words with similar surface forms, including true cognates, false friends, and modern loanwords. These lexical relationships create ambiguity for cross-lingual semantic interpretation, as similar forms may correspond to shared, divergent, or borrowed meanings. To evaluate whether LLMs can make these distinctions, we introduce SemCog Bench, a curated benchmark of 1,858 Arabic--Hebrew word pairs with sentence-level annotations for cognate identification and semantic disambiguation. We evaluate a diverse set of open-source and proprietary LLMs across multiple input representations and contextual settings. Our results show reliance on surface-form similarity, with weaker performance on false friends and wide variation on loanwords. We further find that context and scale yield model-dependent gains, while original-script inputs generally perform best. Our findings reveal limitations in cross-lingual form-meaning reasoning and establish SemCog Bench as a benchmark for cross-lingual lexical semantics. Our code and data are publicly available.