MGAL: A Multilingual Granularity-Aware Long-Context Benchmark
Not on the map. Why?
Abstract (arXiv)
Evaluation of long-context Large Language Models (LLMs) has advanced rapidly. However, most existing benchmarks are limited to the document level and focus mainly on high-resource languages, leaving many fine-grained challenges insufficiently evaluated. To address this gap, we present MGAL, the first multilingual, granularity- and position-aware long-context benchmark. MGAL is constructed from United Nations (UN) reports spanning 8K to 128K tokens across the six official UN languages. It covers four coherent levels of linguistic granularity (word, sentence, paragraph, and document) and further stratifies entries by their position within the document (begin, middle, and end), indexed at both the document and paragraph levels. This design enables systematic diagnosis of multilingual long-context comprehension across different granularities. Through extensive experiments and analyses, we find that: (1) LLMs perform well at word-level tasks but struggle with coarser-grained ones; and (2) Closed-source models retain a clear performance advantage in lower-resource languages. We further identify two new challenges: (1) Under local semantic crowding, where neighboring sentences share topics and entities, models tend to follow surface cues (e.g., connectives like ``however'' or repeated entities) rather than the discourse role of the sentence in surrounding context (e.g., background, outcome); and (2) A gap between fluency and consistency in generated outputs, where models produce text that reads smoothly but drifts from the source facts. In addition, we observe several patterns in line with prior studies, including reliance on nearby evidence and reuse of options under uncertainty.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 38 references that are in the map. First posted on arXiv 21 Aug 2026.
Builds on
22 of 38 references.
References · 22
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
- 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
- 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
- 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
- 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
- 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
- 2017Why We Need New Evaluation Metrics for NLGcited by 233
10 more
- 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
- 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
- 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 90
- 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 81
- 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 57
- 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densitiescited by 32
- 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
- 2025One ruler to measure them all: Benchmarking multilingual long-context language modelscited by 19
- 2023M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Modelscited by 15
- 2024Can we obtain significant success in RST discourse parsing by using Large Language Models?cited by 4
Built on by
Not cited yet.