# CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models

arXiv: https://arxiv.org/abs/2403.03514 · HTML: https://mlsanity.com/paper/2403.03514/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 50 references that are in the map.
- First posted on arXiv 6 Mar 2024.

## Builds on (most cited first, 26 of 26)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [L-Eval: Instituting Standardized Evaluation for Long Context Language Models](https://mlsanity.com/paper/2307.11088/) (2023)
- [ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding](https://mlsanity.com/paper/2305.14196/) (2023)
- [Scaling Laws of RoPE-based Extrapolation](https://mlsanity.com/paper/2310.05209/) (2023)
- [Do Long-Range Language Models Actually Use Long-Range Context?](https://mlsanity.com/paper/2109.09115/) (2021)
- [KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation](https://mlsanity.com/paper/2205.09921/) (2022)
- [A Span-Extraction Dataset for Chinese Machine Reading Comprehension](https://mlsanity.com/paper/1810.07366/) (2018)
- [Unsupervised Text Generation by Learning from Search](https://mlsanity.com/paper/2007.08557/) (2020)
- [Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering](https://mlsanity.com/paper/1607.06275/) (2016)
- [General and Domain Adaptive Chinese Spelling Check with Error Consistent Pretraining](https://mlsanity.com/paper/2203.10929/) (2022)
- [LogiGAN: Learning Logical Reasoning via Adversarial Pre-training](https://mlsanity.com/paper/2205.08794/) (2022)
- [AMERICANO: Argument Generation with Discourse-driven Decomposition and Agent Interaction](https://mlsanity.com/paper/2310.20352/) (2023)

## Built on by (most cited first, 15 of 15)

- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [Who is Introducing the Failure? Automatically Attributing Failures of Multi-Agent Systems via Spectrum Analysis](https://mlsanity.com/paper/2509.13782/) (2025)
- [Entropy-Based Decoding for Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2406.17519/) (2024)
- [LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark](https://mlsanity.com/paper/2601.02872/) (2026)
- [Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMs](https://mlsanity.com/paper/2502.12462/) (2025)
- [Evaluating Multilingual Long-Context Models for Retrieval and Reasoning](https://mlsanity.com/paper/2409.18006/) (2024)
- [SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehension](https://mlsanity.com/paper/2508.01959/) (2025)
- [GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignment](https://mlsanity.com/paper/2605.19577/) (2026)
- [AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection](https://mlsanity.com/paper/2505.07293/) (2025)
- [AIR: Post-training Data Selection for Reasoning via Attention Head Influence](https://mlsanity.com/paper/2512.13279/) (2025)
- [SCALAR: Scientific Citation-based Live Assessment of Long-context Academic Reasoning](https://mlsanity.com/paper/2502.13753/) (2025)
- [LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model](https://mlsanity.com/paper/2511.04952/) (2025)
- [Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction](https://mlsanity.com/paper/2507.16271/) (2025)
- [TPBench: A Turning-Point Benchmark for Dialogue Compression](https://mlsanity.com/paper/2610.02736/) (2026)
- [Delta Tokenizer: Reference-Identical Tokenization of Session Deltas and Segment-Parallelism for Long Prompts on Multicore CPUs](https://mlsanity.com/paper/2610.19801/) (2026)

## Abstract (arXiv)

Developing Large Language Models (LLMs) with robust long-context capabilities has been the recent research focus, resulting in the emergence of long-context LLMs proficient in Chinese. However, the evaluation of these models remains underdeveloped due to a lack of benchmarks. To address this gap, we present CLongEval, a comprehensive Chinese benchmark for evaluating long-context LLMs. CLongEval is characterized by three key features: (1) Sufficient data volume, comprising 7 distinct tasks and 7,267 examples; (2) Broad applicability, accommodating to models with context windows size from 1K to 100K; (3) High quality, with over 2,000 manually annotated question-answer pairs in addition to the automatically constructed labels. With CLongEval, we undertake a comprehensive assessment of 6 open-source long-context LLMs and 2 leading commercial counterparts that feature both long-context abilities and proficiency in Chinese. We also provide in-depth analysis based on the empirical results, trying to shed light on the critical capabilities that present challenges in long-context settings. The dataset, evaluation scripts, and model outputs are released.