mlsanityData to 7 Oct 2026

Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA

arXiv 2406.17419 ↗First posted 25 Jun 2024cs.CL cs.AIcited by 121

Not on the map. Why?

Cited by 25 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Long-context modeling capabilities have garnered widespread attention, leading to the emergence of Large Language Models (LLMs) with ultra-context windows. Meanwhile, benchmarks for evaluating long-context LLMs are gradually catching up. However, existing benchmarks employ irrelevant noise texts to artificially extend the length of test cases, diverging from the real-world scenarios of long-context applications. To bridge this gap, we propose a novel long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA). Unlike typical document QA, in Loong's test cases, each document is relevant to the final answer, ignoring any document will lead to the failure of the answer. Furthermore, Loong introduces four types of tasks with a range of context lengths: Spotlight Locating, Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic and comprehensive evaluation of long-context understanding. Extensive experiments indicate that existing long-context language models still exhibit considerable potential for enhancement. Retrieval augmented generation (RAG) achieves poor performance, demonstrating that Loong can reliably assess the model's long-context modeling capabilities.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

20 of 33 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  10. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  11. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  12. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
8 more
  1. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  2. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  3. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  4. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  5. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  6. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  7. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  8. 2023Giraffe: Adventures in Expanding Context Lengths in LLMscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 64; 2026: 34.

121 papers.

Sort:

By area:

  1. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  2. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 236
  3. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 218
  4. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 144
  5. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  6. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 120
  7. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
  8. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 114
  9. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  10. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  11. 2024StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurizationcited by 70
  12. 2025WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agentscited by 66
38 more
  1. 2025ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agentscited by 64
  2. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  3. 2025WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learningcited by 59
  4. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 47
  5. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generationcited by 40
  6. 2025LongCodeBench: Evaluating Coding LLMs at 1M Context Windowscited by 37
  7. 2024LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarioscited by 31
  8. 2024IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimizationcited by 30
  9. 2025Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunitiescited by 30
  10. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  11. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
  12. 2024Compute Or Load KV Cache? Why Not Both?cited by 25
  13. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answeringcited by 25
  14. 2024AAAR-1.0: Assessing AI's Potential to Assist Researchcited by 24
  15. 2025Hyper-RAG: Combating LLM Hallucinations using Hypergraph-Driven Retrieval-Augmented Generationcited by 23
  16. 2025BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMscited by 22
  17. 2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystackcited by 20
  18. 2025AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented Generationcited by 19
  19. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 17
  20. 2025More Documents, Same Length: Isolating the Challenge of Multiple Documents in RAGcited by 16
  21. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  22. 2025MEBench: Benchmarking Large Language Models for Cross-Document Multi-Entity Question Answeringcited by 15
  23. 2025PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluationcited by 14
  24. 2025LastingBench: Defend Benchmarks Against Knowledge Leakagecited by 14
  25. 2024Towards LifeSpan Cognitive Systemscited by 13
  26. 2024MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMscited by 13
  27. 2025When Refusals Fail: Unstable Safety Mechanisms in Long-Context LLM Agentscited by 13
  28. 2024ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coveragecited by 12
  29. 2024The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trendscited by 11
  30. 2025Eliciting In-context Retrieval and Reasoning for Long-context Large Language Modelscited by 11
  31. 2026LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generationcited by 11
  32. 2024MDCure: A Scalable Pipeline for Multi-Document Instruction-Followingcited by 10
  33. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
  34. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  35. 2025LongAttn: Selecting Long-context Training Data via Token-level Attentioncited by 9
  36. 2025DeepSolution: Boosting Complex Engineering Solution Design via Tree-based Exploration and Bi-point Thinkingcited by 9
  37. 2025Cog-RAG: Cognitive-Inspired Dual-Hypergraph with Theme Alignment Retrieval-Augmented Generationcited by 9
  38. 2025PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attentioncited by 8

50 most cited of 121 All 121 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
  2. 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering
  3. 2025Tagging-Augmented Generation: Assisting Language Models in Finding Intricate Knowledge In Long Contexts
  4. 2023Marathon: A Race Through the Realm of Long Context with Large Language Models
  5. 2025Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMs
  6. 2025Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-k
  7. 2023Retrieval meets Long Context Large Language Models
  8. 2025SagaScale: A Realistic, Scalable, and High-Quality Long-Context Benchmark Built from Full-Length Novels