mlsanityData to 7 Oct 2026

One ruler to measure them all: Benchmarking multilingual long-context language models

arXiv 2503.01996 ↗First posted 3 Mar 2025cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

We present ONERULER, a multilingual benchmark designed to evaluate long-context language models across 26 languages. ONERULER adapts the English-only RULER benchmark (Hsieh et al., 2024) by including seven synthetic tasks that test both retrieval and aggregation, including new variations of the "needle-in-a-haystack" task that allow for the possibility of a nonexistent needle. We create ONERULER through a two-step process, first writing English instructions for each task and then collaborating with native speakers to translate them into 25 additional languages. Experiments with both open-weight and closed LLMs reveal a widening performance gap between low- and high-resource languages as context length increases from 8K to 128K tokens. Surprisingly, English is not the top-performing language on long-context tasks (ranked 6th out of 26), with Polish emerging as the top language. Our experiments also show that many LLMs (particularly OpenAI's o3-mini-high) incorrectly predict the absence of an answer, even in high-resource languages. Finally, in cross-lingual scenarios where instructions and context appear in different languages, performance can fluctuate by up to 20% depending on the instruction language. We hope the release of ONERULER will facilitate future research into improving multilingual and cross-lingual long-context training pipelines.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 32 of its 40 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

32 of 40 references.

References · 32

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  7. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  8. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  9. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  10. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  11. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  12. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
20 more
  1. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  2. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
  3. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  4. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  5. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  6. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  7. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 74
  8. 2024FABLES: Evaluating faithfulness and content selection in book-length summarizationcited by 65
  9. 2022Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP Worldcited by 55
  10. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  11. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 53
  12. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 47
  13. 2024RepoQA: Evaluating Long Context Code Understandingcited by 35
  14. 2025BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Modelscited by 23
  15. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  16. 2024Counting-Stars: A Multi-evidence, Position-aware, and Scalable Benchmark for Evaluating Long-Context Large Language Modelscited by 17
  17. 2024Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contextscited by 14
  18. 2024The Zeno's Paradox of `Low-Resource' Languagescited by 12
  19. 2024Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Modelscited by 9
  20. 2024Evaluating Multilingual Long-Context Models for Retrieval and Reasoningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 10.

18 papers.

Sort:

By area:

  1. 2025DeepSeek-R1 Thoughtology: Let's think about LLM Reasoningcited by 91
  2. 2025Does quantization affect models' performance on long-context tasks?cited by 10
  3. 2025Code Like Humans: A Multi-Agent Solution for Medical Codingcited by 10
  4. 2026Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptationcited by 4
  5. 2025OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviewscited by 3
  6. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 3
  7. 2025Harnessing the Unseen: The Hidden Influence of Intrinsic Knowledge in Long-Context Language Modelscited by 1
  8. 2025LoopServe: An Adaptive Dual-phase LLM Inference Acceleration System for Multi-Turn Dialoguescited by 1
  9. 2026Multilingual Large Language Models do not comprehend all natural languages to equal degreescited by 1
  10. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 1
  11. 2025Quantification and object perception in Multimodal Large Language Models and human linguistic cognitioncited by 0
  12. 2026XLGoBench: Detecting cross-lingual skill gaps with algorithmic taskscited by 0
6 more
  1. 2026Dense Contexts Are Hard Contexts: Lexical Density Limits Effective Context in LLMscited by 0
  2. 2026NarrativeWorldBench: A Frontier-Saturated Benchmark and a Latent World Model for Long-Horizon Co-Creative Audio Dramacited by 0
  3. 2026Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+cited by 0
  4. 2026MGAL: A Multilingual Granularity-Aware Long-Context Benchmarkcited by 0
  5. 2026The Limits of BPE Tokenization in Polish: Segmentation-Flexional Forms, Grammatical Anchoring, and First-Person Stability in Inflectional Language Modelscited by 0
  6. 2026Multilinguality in Hybrid Attention LLMscited by 0