mlsanityData to 7 Oct 2026

Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems

arXiv 2407.01370 ↗First posted 1 Jul 2024cs.CLcited by 87

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

LLMs and RAG systems are now capable of handling millions of input tokens or more. However, evaluating the output quality of such systems on long-context tasks remains challenging, as tasks like Needle-in-a-Haystack lack complexity. In this work, we argue that summarization can play a central role in such evaluation. We design a procedure to synthesize Haystacks of documents, ensuring that specific textitinsights repeat across documents. The "Summary of a Haystack" (SummHay) task then requires a system to process the Haystack and generate, given a query, a summary that identifies the relevant insights and precisely cites the source documents. Since we have precise knowledge of what insights should appear in a haystack summary and what documents should be cited, we implement a highly reproducible automatic evaluation that can score summaries on two aspects - Coverage and Citation. We generate Haystacks in two domains (conversation, news), and perform a large-scale evaluation of 10 LLMs and corresponding 50 RAG systems. Our findings indicate that SummHay is an open challenge for current systems, as even systems provided with an Oracle signal of document relevance lag our estimate of human performance (56%) by 10+ points on a Joint Score. Without a retriever, long-context LLMs like GPT-4o and Claude 3 Opus score below 20% on SummHay. We show SummHay can also be used to study enterprise RAG systems and position bias in long-context models. We hope future systems can equal and surpass human performance on SummHay.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 43 of its 49 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

43 of 49 references.

References · 43

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  12. 2015Teaching Machines to Read and Comprehendcited by 1,039
31 more
  1. 2019ELI5: Long Form Question Answeringcited by 567
  2. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  3. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  4. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  5. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  6. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  7. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  8. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  9. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  10. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  11. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  12. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  13. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86
  14. 2023A Survey of Large Language Models Attributioncited by 71
  15. 2024FABLES: Evaluating faithfulness and content selection in book-length summarizationcited by 65
  16. 2023HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attributioncited by 55
  17. 2023Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articlescited by 45
  18. 2023Understanding Retrieval Augmentation for Long-Form Question Answeringcited by 45
  19. 2021Exploring Neural Models for Query-Focused Summarizationcited by 44
  20. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  21. 2023Less is More for Long Document Summary Evaluation by LLMscited by 28
  22. 2021The Summary Loop: Learning to Write Abstractive Summaries Without Examplescited by 27
  23. 2024AttributionBench: How Hard is Automatic Attribution Evaluation?cited by 26
  24. 2023Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluationcited by 24
  25. 2023Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attributioncited by 24
  26. 2024Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Taskscited by 23
  27. 2024Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphscited by 16
  28. 2023M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Modelscited by 15
  29. 2024LLM In-Context Recall is Prompt Dependentcited by 15
  30. 2024XL2Bench: A Benchmark for Extremely Long Context Understanding with Long-range Dependenciescited by 14
  31. 2022Discord Questions: A Computational Approach To Diversity Analysis in News Coveragecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 47; 2026: 19.

87 papers.

Sort:

By area:

  1. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  2. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  3. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  4. 2024CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environmentscited by 69
  5. 2024The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Modelscited by 67
  6. 2025Glyph: Scaling Context Windows via Visual-Text Compressioncited by 49
  7. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28
  8. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  9. 2024Long Context RAG Performance of Large Language Modelscited by 23
  10. 2024SiReRAG: Indexing Similar and Related Information for Multihop Reasoningcited by 23
  11. 2025Benchmarking Deep Search over Heterogeneous Enterprise Datacited by 23
  12. 2024On Positional Bias of Faithfulness for Long-form Summarizationcited by 22
38 more
  1. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 22
  2. 2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystackcited by 20
  3. 2025One ruler to measure them all: Benchmarking multilingual long-context language modelscited by 18
  4. 2025GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluationcited by 18
  5. 2024From Single to Multi: How LLMs Hallucinate in Multi-Document Summarizationcited by 17
  6. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  7. 2024Perceptions of Linguistic Uncertainty by Language Models and Humanscited by 15
  8. 2024Search Engines in an AI Era: The False Promise of Factual and Verifiable Source-Cited Responsescited by 15
  9. 2024MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMscited by 13
  10. 2025A Dataset for Addressing Patient's Information Needs related to Clinical Course of Hospitalizationcited by 13
  11. 2024Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?cited by 12
  12. 2024Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Datacited by 11
  13. 2025Behavioral Analysis of Information Salience in Large Language Modelscited by 11
  14. 2024Understanding Synthetic Context Extension via Retrieval Headscited by 10
  15. 2025RAG-KG-IL: A Multi-Agent Hybrid Framework for Reducing Hallucinations and Enhancing LLM Reasoning through RAG and Incremental Knowledge Graph Learning Integrationcited by 9
  16. 2026LLMs Corrupt Your Documents When You Delegatecited by 9
  17. 2024Introducing a new hyper-parameter for RAG: Context Window Utilizationcited by 8
  18. 2024Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coveragecited by 8
  19. 2024Truth or Mirage? Towards End-to-End Factuality Evaluation with LLM-Oasiscited by 8
  20. 2025EviNote-RAG: Enhancing RAG Models via Answer-Supportive Evidence Notescited by 8
  21. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  22. 2025Evaluating the Effect of Retrieval Augmentation on Social Biasescited by 7
  23. 2025MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Modelscited by 7
  24. 2025Attribution, Citation, and Quotation: A Survey of Evidence-based Text Generation with Large Language Modelscited by 7
  25. 2025PRISM: Agentic Retrieval with LLMs for Multi-Hop Question Answeringcited by 7
  26. 2026Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interactioncited by 7
  27. 2025Unstructured Evidence Attribution for Long Context Query Focused Summarizationcited by 5
  28. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  29. 2025GenerationPrograms: Fine-grained Attribution with Executable Programscited by 4
  30. 2025Memory Limitations of Prompt Tuning in Transformerscited by 4
  31. 2025Rethinking Schema Linking: A Context-Aware Bidirectional Retrieval Approach for Text-to-SQLcited by 4
  32. 2024Retrieval Or Holistic Understanding? Dolce: Differentiate Our Long Context Evaluation Taskscited by 3
  33. 2024DeFine: Decision-Making with Analogical Reasoning over Factor Profilescited by 3
  34. 2025Scaling Multi-Document Event Summarization: Evaluating Compression vs. Full-Text Approachescited by 3
  35. 2025Lost-in-the-Middle in Long-Text Generation: Synthetic Dataset, Evaluation Framework, and Mitigationcited by 3
  36. 2025Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysiscited by 3
  37. 2025BEE-RAG: Balanced Entropy Engineering for Retrieval-Augmented Generationcited by 3
  38. 2025OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviewscited by 3

50 most cited of 87 All 87 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Tree Navigation Without LLM Summaries: A Matched-Cost Study of Hierarchical Retrieval for Long-Document QA
  2. 2026sui-1: Grounded and Verifiable Long-Form Summarization
  3. 2024L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding?
  4. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models
  5. 2025Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languages
  6. 2024UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis
  7. 2025Rethinking All Evidence: Enhancing Trustworthy Retrieval-Augmented Generation via Conflict-Driven Summarization
  8. 2024Rethinking Scientific Summarization Evaluation: Grounding Explainable Metrics on Facet-aware Benchmark