mlsanityData to 14 Oct 2026

∞Bench: Extending Long Context Evaluation Beyond 100K Tokens

arXiv 2402.13718 ↗First posted 21 Feb 2024cs.CLcited by 367

Not on the map. Why?

Cited by 64 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Processing and reasoning over long contexts is crucial for many practical applications of Large Language Models (LLMs), such as document comprehension and agent construction. Despite recent strides in making LLMs process contexts with more than 100K tokens, there is currently a lack of a standardized benchmark to evaluate this long-context capability. Existing public benchmarks typically focus on contexts around 10K tokens, limiting the assessment and comparison of LLMs in processing longer contexts. In this paper, we propose ∞Bench, the first LLM benchmark featuring an average data length surpassing 100K tokens. ∞Bench comprises synthetic and realistic tasks spanning diverse domains, presented in both English and Chinese. The tasks in ∞Bench are designed to require well understanding of long dependencies in contexts, and make simply retrieving a limited number of passages from contexts not sufficient for these tasks. In our experiments, based on ∞Bench, we evaluate the state-of-the-art proprietary and open-source LLMs tailored for processing long contexts. The results indicate that existing long context LLMs still require significant advancements to effectively process 100K+ context. We further present three intriguing analyses regarding the behavior of LLMs processing long context.

Key facts. Cited by 367 papers in the map (data through 14 Oct 2026). Builds on 40 of its 53 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

40 of 53 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  12. 2020Longformer: The Long-Document Transformercited by 3,413
28 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  11. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  12. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  13. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  14. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  15. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  16. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  17. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  18. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  19. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  20. 2021Efficient Attentions for Long Document Summarizationcited by 345
  21. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  22. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  23. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  24. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  25. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  26. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  27. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 71
  28. 2023A Survey on Long Text Modeling with Transformerscited by 47

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 88; 2025: 148; 2026: 131.

367 papers.

Sort:

By area:

  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  2. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  3. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  4. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  5. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
  6. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  7. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 281
  8. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  9. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  10. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 183
  11. 2025Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scalecited by 178
  12. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 148
38 more
  1. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  2. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
  3. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 127
  4. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 126
  5. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  6. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123
  7. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  8. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
  9. 2024Make Your LLM Fully Utilize the Contextcited by 119
  10. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 119
  11. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentationcited by 119
  12. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  13. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 114
  14. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 97
  15. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  16. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 95
  17. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 86
  18. 2026MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Taskscited by 84
  19. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  20. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 69
  21. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 63
  22. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  23. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 60
  24. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 60
  25. 2025Mem-α: Learning Memory Construction via Reinforcement Learningcited by 55
  26. 2025M+: Extending MemoryLLM with Scalable Long-Term Memorycited by 54
  27. 2024ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilitiescited by 53
  28. 2024In Defense of RAG in the Era of Long-Context Language Modelscited by 53
  29. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 50
  30. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  31. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 42
  32. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 40
  33. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  34. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 39
  35. 2024HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systemscited by 37
  36. 2025LongCodeBench: Evaluating Coding LLMs at 1M Context Windowscited by 37
  37. 2024Insights into LLM Long-Context Failures: When Transformers Know but Don't Tellcited by 36
  38. 2026From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agentscited by 36

50 most cited of 367 All 367 (JSON)