# XL2Bench: A Benchmark for Extremely Long Context Understanding with Long-range Dependencies

arXiv: https://arxiv.org/abs/2404.05446 · HTML: https://mlsanity.com/paper/2404.05446/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2024.

## Built on by (most cited first, 14 of 14)

- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption](https://mlsanity.com/paper/2407.18003/) (2024)
- [Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems](https://mlsanity.com/paper/2407.01370/) (2024)
- [LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios](https://mlsanity.com/paper/2411.07037/) (2024)
- [A Survey on Transformer Context Extension: Approaches and Evaluation](https://mlsanity.com/paper/2503.13299/) (2025)
- [PRELUDE: A Benchmark Designed to Require Global Comprehension and Reasoning over Long Contexts](https://mlsanity.com/paper/2508.09848/) (2025)
- [Can AI Validate Science? Benchmarking LLMs for Accurate Scientific Claim rightarrow Evidence Reasoning](https://mlsanity.com/paper/2506.08235/) (2025)
- [Divide, Optimize, Merge: Fine-Grained LLM Agent Optimization at Scale](https://mlsanity.com/paper/2505.03973/) (2025)
- [LongEmotion: Measuring Emotional Intelligence of Large Language Models in Long-Context Interaction](https://mlsanity.com/paper/2509.07403/) (2025)
- [CNNSum: Exploring Long-Context Summarization with Large Language Models in Chinese Novels](https://mlsanity.com/paper/2412.02819/) (2024)
- [Respecting Temporal-Causal Consistency: Entity-Event Knowledge Graphs for Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.05939/) (2025)
- [OPOR-Bench: Evaluating Large Language Models on Online Public Opinion Report Generation](https://mlsanity.com/paper/2512.01896/) (2025)
- [Context-Length Robustness in Question Answering Models: A Comparative Empirical Study](https://mlsanity.com/paper/2603.15723/) (2026)
- [EvalMem: An Operation-Level Diagnostic Framework for Long-Term Memory Systems](https://mlsanity.com/paper/2609.22231/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable performance across diverse tasks but are constrained by their small context window sizes. Various efforts have been proposed to expand the context window to accommodate even up to 200K input tokens. Meanwhile, building high-quality benchmarks with much longer text lengths and more demanding tasks to provide comprehensive evaluations is of immense practical interest to facilitate long context understanding research of LLMs. However, prior benchmarks create datasets that ostensibly cater to long-text comprehension by expanding the input of traditional tasks, which falls short to exhibit the unique characteristics of long-text understanding, including long dependency tasks and longer text length compatible with modern LLMs' context window size. In this paper, we introduce a benchmark for extremely long context understanding with long-range dependencies, XL2Bench, which includes three scenarios: Fiction Reading, Paper Reading, and Law Reading, and four tasks of increasing complexity: Memory Retrieval, Detailed Understanding, Overall Understanding, and Open-ended Generation, covering 27 subtasks in English and Chinese. It has an average length of 100K+ words (English) and 200K+ characters (Chinese). Evaluating six leading LLMs on XL2Bench, we find that their performance significantly lags behind human levels. Moreover, the observed decline in performance across both the original and enhanced datasets underscores the efficacy of our approach to mitigating data contamination.