mlsanityData to 7 Oct 2026

LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

arXiv 2308.14508 ↗First posted 28 Aug 2023cs.CLcited by 1,378

Not on the map. Why?

Cited by 460 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Although large language models (LLMs) demonstrate impressive performance for many language tasks, most of them can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases. Recent works have proposed methods to improve LLMs' long context capabilities by extending context windows and more sophisticated memory mechanisms. However, comprehensive benchmarks tailored for evaluating long context understanding are lacking. In this paper, we introduce LongBench, the first bilingual, multi-task benchmark for long context understanding, enabling a more rigorous evaluation of long context understanding. LongBench comprises 21 datasets across 6 task categories in both English and Chinese, with an average length of 6,711 words (English) and 13,386 characters (Chinese). These tasks cover key long-text application areas including single-doc QA, multi-doc QA, summarization, few-shot learning, synthetic tasks, and code completion. All datasets in LongBench are standardized into a unified format, allowing for effortless automatic evaluation of LLMs. Upon comprehensive evaluation of 8 LLMs on LongBench, we find that: (1) Commercial model (GPT-3.5-Turbo-16k) outperforms other open-sourced models, but still struggles on longer contexts. (2) Scaled position embedding and fine-tuning on longer sequences lead to substantial improvement on long context understanding. (3) Context compression technique such as retrieval brings improvement for model with weak ability on long contexts, but the performance still lags behind models that have strong long context understanding capability. The code and datasets are available at https://github.com/THUDM/LongBench.

Key facts. Cited by 1,378 papers in the map (data through 7 Oct 2026). Builds on 56 of its 69 references that are in the map. First posted on arXiv 28 Aug 2023.

Builds on

56 of 69 references.

References · 56

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
38 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  2. 2020Reformer: The Efficient Transformercited by 1,658
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  4. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  6. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  7. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  10. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  11. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  12. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  13. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  14. 2020Efficient Transformers: A Surveycited by 784
  15. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  16. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  17. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  18. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  19. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  20. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  21. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  22. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  23. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  24. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  25. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  26. 2021Efficient Attentions for Long Document Summarizationcited by 342
  27. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  28. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  29. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  30. 2022Memorizing Transformerscited by 219
  31. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  32. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  33. 2022Recurrent Memory Transformercited by 197
  34. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  35. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  36. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  37. 2022A Length-Extrapolatable Transformercited by 123
  38. 2017DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applicationscited by 110

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 28; 2024: 294; 2025: 458; 2026: 598.

1,378 papers.

Sort:

By area:

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  4. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  7. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  8. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  9. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  10. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  11. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  12. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
38 more
  1. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  2. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  3. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  4. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  5. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  6. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 236
  7. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  8. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  9. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  10. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  11. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  12. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  13. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  14. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  15. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  16. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  17. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  18. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  19. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  20. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  21. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  22. 2025Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scalecited by 163
  23. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  24. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
  25. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 142
  26. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  27. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  28. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  29. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  30. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  31. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  32. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  33. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 121
  34. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 121
  35. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  36. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  37. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 120
  38. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 120

50 most cited of 1,378 All 1,378 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023CMMLU: Measuring massive multitask language understanding in Chinese
  2. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
  3. 2025Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
  4. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation
  5. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models
  6. 2024An Effective Framework to Help Large Language Models Handle Numeric-involved Long-context Tasks
  7. 2026Retrieval and Multi-Hop Reasoning in 1M-Token Context Windows: Evaluating LLMs on Classical Chinese Text
  8. 2024MileBench: Benchmarking MLLMs in Long Context