mlsanityData to 7 Oct 2026

Evaluating Multilingual Long-Context Models for Retrieval and Reasoning

arXiv 2409.18006 ↗First posted 26 Sep 2024cs.CLcited by 8

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Recent large language models (LLMs) demonstrate impressive capabilities in handling long contexts, some exhibiting near-perfect recall on synthetic retrieval tasks. However, these evaluations have mainly focused on English text and involved a single target sentence within lengthy contexts. Our work investigates how LLM performance generalizes to multilingual settings with multiple hidden target sentences. We create a new dataset -- mLongRR -- to comprehensively evaluate several multilingual long-context LLMs on retrieval and reasoning tasks across five languages: English, Vietnamese, Indonesian, Swahili, and Somali. These languages share the Latin script but belong to distinct language families and resource levels. Our analysis reveals a significant performance gap between languages. The best-performing models such as Gemini-1.5 and GPT-4o, achieve around 96% accuracy in English to around 36% in Somali with a single target sentence. However, this accuracy drops to 40% in English and 0% in Somali when dealing with three target sentences. Our findings highlight the challenges long-context LLMs face when processing longer contexts, an increase in the number of target sentences, or languages of lower resource levels.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 12 of its 35 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

12 of 35 references.

References · 12

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  6. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  7. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  8. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  9. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  10. 2023Giraffe: Adventures in Expanding Context Lengths in LLMscited by 32
  11. 2024CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Modelscited by 13
  12. 2023Exploring the Maze of Multilingual Modelingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 2.

8 papers.

  1. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
  2. 2025A Survey of Multilingual Reasoning in Language Modelscited by 31
  3. 2025One ruler to measure them all: Benchmarking multilingual long-context language modelscited by 18
  4. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  5. 2025Enhancing Large Language Models with Neurosymbolic Reasoning for Multilingual Taskscited by 4
  6. 2025On the Sensitivity of Instruction-tuned LLMs to Harmful Sentences in Long Inputscited by 0
  7. 2026Signals Are Not States: Neuro-Symbolic Safeguards for Culturally Aware Classroom AIcited by 0
  8. 2026Choosing the Right Language Mode at Inference Time for Multilingual Reliabilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
  2. 2023Retrieval meets Long Context Large Language Models
  3. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation
  4. 2024Long-context Language Models Fail in Basic Retrieval Tasks Without Sufficient Reasoning Steps
  5. 2025LC-Eval: A Bilingual Multi-Task Evaluation Benchmark for Long-Context Understanding
  6. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
  7. 2025On the Consistency of Multilingual Context Utilization in Retrieval-Augmented Generation
  8. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models