mlsanityData to 7 Oct 2026

BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models

arXiv 2309.13345 ↗First posted 23 Sep 2023cs.CLcited by 53

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have achieved dramatic proficiency over NLP tasks with normal length. Recently, multiple studies have committed to extending the context length and enhancing the long text modeling capabilities of LLMs. To comprehensively evaluate the long context ability of LLMs, we propose BAMBOO, a multi-task long context benchmark. BAMBOO has been designed with four principles: comprehensive capacity evaluation, avoidance of data contamination, accurate automatic evaluation, and different length levels. It consists of 10 datasets from 5 different long text understanding tasks, i.e. question answering, hallucination detection, text sorting, language modeling, and code completion, to cover core capacities and various domains of LLMs. We conduct experiments with five long context models on BAMBOO and further discuss four key research questions of long text. We also qualitatively analyze current long context models and point out future directions for enhancing long text modeling capacities. We release our data, prompts, and code at https://github.com/RUCAIBox/BAMBOO.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 62 of its 77 references that are in the map. First posted on arXiv 23 Sep 2023.

Builds on

62 of 77 references.

References · 62

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
38 more
  1. 2020Longformer: The Long-Document Transformercited by 3,362
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  13. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  14. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  15. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  16. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  17. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  18. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  19. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  20. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  21. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  22. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  23. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  24. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  25. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  26. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  27. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  28. 2021Random Feature Attentioncited by 350
  29. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  30. 2021Efficient Attentions for Long Document Summarizationcited by 342
  31. 2022Long Range Language Modeling via Gated State Spacescited by 265
  32. 2020Sparse Sinkhorn Attentioncited by 250
  33. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  34. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  35. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  36. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
  37. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  38. 2022Dense Text Retrieval based on Pretrained Language Models: A Surveycited by 136

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 23; 2025: 19; 2026: 10.

53 papers.

Sort:

By area:

  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  3. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  4. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  5. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 116
  6. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  7. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  8. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  9. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 47
  10. 2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokenscited by 31
  11. 2025CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoningcited by 28
  12. 2024Exploring Large Language Models for Feature Selection: A Data-centric Perspectivecited by 27
38 more
  1. 2024Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Modelscited by 20
  2. 2025StoryBench: A Dynamic Benchmark for Evaluating Long-Term Memory with Multi Turnscited by 20
  3. 2024Exploring Context Window of Large Language Models via Decomposed Positional Vectorscited by 18
  4. 2025One ruler to measure them all: Benchmarking multilingual long-context language modelscited by 18
  5. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understandingcited by 16
  6. 2023ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?cited by 15
  7. 2024MedOdyssey: A Medical Domain Benchmark for Long Context Evaluation Up to 200K Tokenscited by 13
  8. 2024MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMscited by 13
  9. 2025Mamba Modulation: On the Length Generalization of Mambacited by 13
  10. 2024Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?cited by 12
  11. 2025Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMscited by 11
  12. 2024ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Modelscited by 9
  13. 2024ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Modelscited by 9
  14. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
  15. 2026AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rolloutscited by 8
  16. 2026CL-bench Life: Can Language Models Learn from Real-Life Context?cited by 8
  17. 2024A Survey of Automatic Hallucination Evaluation on Natural Language Generationcited by 7
  18. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decodingcited by 7
  19. 2024LIFT: Improving Long Context Understanding Through Long Input Fine-Tuningcited by 6
  20. 2025Single-Pass Document Scanning for Question Answeringcited by 6
  21. 2024L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding?cited by 5
  22. 2024Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidencecited by 5
  23. 2025WikiAutoGen: Towards Multi-Modal Wikipedia-Style Article Generationcited by 5
  24. 2026Metis: Memory Foundation Modelcited by 5
  25. 2025Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluationcited by 4
  26. 2025AcademicEval: Live Long-Context LLM Benchmarkcited by 4
  27. 2026Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancingcited by 4
  28. 2024Retrieval Or Holistic Understanding? Dolce: Differentiate Our Long Context Evaluation Taskscited by 3
  29. 2025Detecting LLM Fact-conflicting Hallucinations Enhanced by Temporal-logic-based Reasoningcited by 2
  30. 2025SCALAR: Scientific Citation-based Live Assessment of Long-context Academic Reasoningcited by 1
  31. 2025Attentive Reasoning Queries: A Systematic Method for Optimizing Instruction-Following in Large Language Modelscited by 1
  32. 2025Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functionscited by 1
  33. 2026Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicitycited by 1
  34. 2026Decoupled Reasoning with Implicit Fact Tokens (DRIFT): A Dual-Model Framework for Efficient Long-Context Inferencecited by 1
  35. 2026KV Cache Offloading for Context-Intensive Taskscited by 1
  36. 2024Leveraging Large Language Models for Comparative Literature Summarization with Reflective Incremental Mechanismscited by 0
  37. 2025Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extractioncited by 0
  38. 2025LLM-FS-Agent: A Deliberative Role-based Large Language Model Architecture for Transparent Feature Selectioncited by 0

50 most cited of 53 All 53 (JSON)