mlsanityData to 7 Oct 2026

BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack

arXiv 2406.10149 ↗First posted 14 Jun 2024cs.CL cs.AIcited by 211

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In recent years, the input context sizes of large language models (LLMs) have increased dramatically. However, existing evaluation methods have not kept pace, failing to comprehensively assess the efficiency of models in handling long contexts. To bridge this gap, we introduce the BABILong benchmark, designed to test language models' ability to reason across facts distributed in extremely long documents. BABILong includes a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets. These tasks are challenging on their own, and even more demanding when the required facts are scattered across long natural text. Our evaluations show that popular LLMs effectively utilize only 10-20% of the context and their performance declines sharply with increased reasoning complexity. Among alternatives to in-context reasoning, Retrieval-Augmented Generation methods achieve a modest 60% accuracy on single-fact question answering, independent of context length. Among context extension methods, the highest performance is demonstrated by recurrent memory transformers after fine-tuning, enabling the processing of lengths up to 50 million tokens. The BABILong benchmark is extendable to any length to support the evaluation of new upcoming models with increased capabilities, and we provide splits up to 10 million token lengths.

Key facts. Cited by 211 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 64 of its 94 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

64 of 94 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Marathon: A Race Through the Realm of Long Context with Large Language Models15 Dec
  2. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  3. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep

Other references · 61

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2023Mistral 7Bcited by 3,065
  8. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  11. 2024Mixtral of Expertscited by 1,785
  12. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
38 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  2. 2018Datasheets for Datasetscited by 1,371
  3. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  6. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  7. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  8. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  9. 2024Yi: Open Foundation Models by 01.AIcited by 706
  10. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  11. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  12. 2014Neural Turing Machinescited by 630
  13. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  14. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  15. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  16. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  17. 2023Adapting Language Models to Compress Contextscited by 350
  18. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  19. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  20. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  21. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  22. 2020Transformers as Soft Reasoners over Languagecited by 265
  23. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  24. 2022Memorizing Transformerscited by 219
  25. 2024Many-Shot In-Context Learningcited by 203
  26. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  27. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  28. 2022Recurrent Memory Transformercited by 197
  29. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  30. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  31. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  32. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  33. 2022Block-Recurrent Transformerscited by 107
  34. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  35. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  36. 2020MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioningcited by 79
  37. 2020Memformer: A Memory-Augmented Transformer for Sequence Modelingcited by 63
  38. 2024DocFinQA: A Long-Context Financial Reasoning Datasetcited by 59

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 88; 2026: 95.

211 papers.

Sort:

By area:

  1. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  2. 2024Titans: Learning to Memorize at Test Timecited by 303
  3. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 142
  4. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 120
  5. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
  6. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  7. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 85
  8. 2024StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurizationcited by 70
  9. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  10. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  11. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Modelscited by 53
  12. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
38 more
  1. 2025TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Recordscited by 33
  2. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 33
  3. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  4. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 32
  5. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 32
  6. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answeringcited by 25
  7. 2024Star Attention: Efficient LLM Inference over Long Sequencescited by 25
  8. 2024AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attributioncited by 23
  9. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 22
  10. 2024DetectiveQA: Evaluating Long-Context Reasoning on Detective Novelscited by 21
  11. 2025Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervisioncited by 20
  12. 2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystackcited by 20
  13. 2025DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversalcited by 20
  14. 2025StoryBench: A Dynamic Benchmark for Evaluating Long-Term Memory with Multi Turnscited by 20
  15. 2025AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented Generationcited by 19
  16. 2025Understanding and Improving Length Generalization in Recurrent Modelscited by 19
  17. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 19
  18. 2025One ruler to measure them all: Benchmarking multilingual long-context language modelscited by 18
  19. 2025Minions: Cost-efficient Collaboration Between On-device and Cloud Language Modelscited by 17
  20. 2025LongFuncEval: Measuring the effectiveness of long context models for function callingcited by 17
  21. 2025Sculptor: Empowering LLMs with Cognitive Agency via Active Context Managementcited by 17
  22. 2025GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?cited by 16
  23. 2024GLIDER: Grading LLM Interactions and Decisions using Explainable Rankingcited by 14
  24. 2025Episodic Memories Generation and Evaluation Benchmark for Large Language Modelscited by 14
  25. 2024ReAttention: Training-Free Infinite Context with Finite Attention Scopecited by 13
  26. 2024MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMscited by 13
  27. 2025LM2: Large Memory Modelscited by 13
  28. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  29. 2025LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformerscited by 12
  30. 2026Memory Caching: RNNs with Growing Memorycited by 12
  31. 2024Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Datacited by 11
  32. 2025TokenButler: Token Importance is Predictablecited by 11
  33. 2025Spatial Traces: Enhancing VLA Models with Spatial-Temporal Understandingcited by 11
  34. 2025Short window attention enables long-term memorizationcited by 11
  35. 2026MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systemscited by 11
  36. 2026Understanding LoRA as Knowledge Memory: An Empirical Analysiscited by 11
  37. 2024On the Benchmarking of LLMs for Open-Domain Dialogue Evaluationcited by 10
  38. 2025Multi-Token Attentioncited by 10

50 most cited of 211 All 211 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
  2. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion
  3. 2025Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
  4. 2024Large Language Models Can Self-Improve in Long-context Reasoning
  5. 2026Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench
  6. 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities
  7. 2025LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data
  8. 2025Improving Context Fidelity via Native Retrieval-Augmented Reasoning