# LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark

arXiv: https://arxiv.org/abs/2601.02872 · HTML: https://mlsanity.com/paper/2601.02872/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Builds on (most cited first, 18 of 18)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [L-Eval: Instituting Standardized Evaluation for Long Context Language Models](https://mlsanity.com/paper/2307.11088/) (2023)
- [ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding](https://mlsanity.com/paper/2305.14196/) (2023)
- [SCROLLS: Standardized CompaRison Over Long Language Sequences](https://mlsanity.com/paper/2201.03533/) (2022)
- [HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly](https://mlsanity.com/paper/2410.02694/) (2024)
- [DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning](https://mlsanity.com/paper/2511.22570/) (2025)
- [Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks](https://mlsanity.com/paper/2404.06480/) (2024)
- [CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models](https://mlsanity.com/paper/2403.03514/) (2024)

## Built on by (most cited first, 19 of 19)

- [Prime Agent: A Self-Improving RLM Harness](https://mlsanity.com/paper/2608.23552/) (2026)
- [GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignment](https://mlsanity.com/paper/2605.19577/) (2026)
- [The Efficiency Frontier: A Unified Framework for Cost-Performance Optimization in LLM Context Management](https://mlsanity.com/paper/2605.23071/) (2026)
- [HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference](https://mlsanity.com/paper/2601.13684/) (2026)
- [VERA: Identifying and Leveraging Visual Evidence Retrieval Heads in Long-Context Understanding](https://mlsanity.com/paper/2602.10146/) (2026)
- [T2S-Bench & Structure-of-Thought: Benchmarking and Prompting Comprehensive Text-to-Structure Reasoning](https://mlsanity.com/paper/2603.03790/) (2026)
- [Understanding LLM Performance Degradation in Multi-Instance Processing: The Roles of Instance Count and Context Length](https://mlsanity.com/paper/2603.22608/) (2026)
- [UNIQUE: Universal Top-k Sparse Attention for Training-free Inference and Sparsity-aware Training](https://mlsanity.com/paper/2605.27740/) (2026)
- [Probing How Scalable Table Data Enhances General Long-Context Reasoning](https://mlsanity.com/paper/2603.21719/) (2026)
- [GraphLit: Learning Text-Enriched Dynamic Character Network Representations for Literary Study](https://mlsanity.com/paper/2605.28643/) (2026)
- [Self-Guided Test-Time Training for Long-Context LLMs](https://mlsanity.com/paper/2607.09415/) (2026)
- [CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference](https://mlsanity.com/paper/2609.26300/) (2026)
- [WildTrace: Benchmarking Natural Evidence Trails in Long-Context Reasoning](https://mlsanity.com/paper/2607.09328/) (2026)
- [Nanbeige4.2-3B on Apple Silicon: Fixing Deployment Bugs and Decreasing Looped Transformer Memory Overhead](https://mlsanity.com/paper/2608.13987/) (2026)
- [Self-Indexing Attention for Compression-Compatible Sparse Long-Context LLM Inference](https://mlsanity.com/paper/2609.13205/) (2026)
- [MWE-ECL: Recoverable Long-Range Context Does Not Always Override Local Lexical Priors](https://mlsanity.com/paper/2609.27590/) (2026)
- [No More Free Lunch: Corpus Task Complexity Matters as Corpora Grow](https://mlsanity.com/paper/2609.29245/) (2026)
- [KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse](https://mlsanity.com/paper/2610.13060/) (2026)
- [AlignPred: Error-Controlled KV Cache Compression via Alignment and Predictive Coding](https://mlsanity.com/paper/2610.17970/) (2026)

## Abstract (arXiv)

The rapid expansion of context length in large language models (LLMs) has outpaced existing evaluation benchmarks. Current long-context benchmarks often trade off scalability and realism: synthetic tasks underrepresent real-world complexity, while fully manual annotation is costly to scale to extreme lengths and diverse scenarios. We present LongBench Pro, a more realistic and comprehensive bilingual benchmark of 1,500 naturally occurring long-context samples in English and Chinese spanning 11 primary tasks and 25 secondary tasks, with input lengths from 8k to 256k tokens. LongBench Pro supports fine-grained analysis with task-specific metrics and a multi-dimensional taxonomy of context requirement (full vs. partial dependency), length (six levels), and difficulty (four levels calibrated by model performance). To balance quality with scalability, we propose a Human-Model Collaborative Construction pipeline: frontier LLMs draft challenging questions and reference answers, along with design rationales and solution processes, to reduce the cost of expert verification. Experts then rigorously validate correctness and refine problematic cases. Evaluating 46 widely used long-context LLMs on LongBench Pro yields three findings: (1) long-context optimization contributes more to long-context comprehension than parameter scaling; (2) effective context length is typically shorter than the claimed context length, with pronounced cross-lingual misalignment; and (3) the "thinking" paradigm helps primarily models trained with native reasoning, while mixed-thinking designs offer a promising Pareto trade-off. In summary, LongBench Pro provides a robust testbed for advancing long-context understanding.