mlsanityData to 13 Oct 2026

Long2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall

arXiv 2410.23000 ↗First posted 30 Oct 2024cs.CLcited by 32

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-augmented generation (RAG) is a promising approach to address the limitations of fixed knowledge in large language models (LLMs). However, current benchmarks for evaluating RAG systems suffer from two key deficiencies: (1) they fail to adequately measure LLMs' capability in handling long-context retrieval due to a lack of datasets that reflect the characteristics of retrieved documents, and (2) they lack a comprehensive evaluation method for assessing LLMs' ability to generate long-form responses that effectively exploits retrieved information. To address these shortcomings, we introduce the Long2RAG benchmark and the Key Point Recall (KPR) metric. Long2RAG comprises 280 questions spanning 10 domains and across 8 question categories, each associated with 5 retrieved documents with an average length of 2,444 words. KPR evaluates the extent to which LLMs incorporate key points extracted from the retrieved documents into their generated responses, providing a more nuanced assessment of their ability to exploit retrieved information.

Key facts. Cited by 32 papers in the map (data through 13 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 28 of its 44 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

28 of 44 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024The Chronicles of RAG: The Retriever, the Chunk and the Generator15 Jan
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  3. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov

Other references · 25

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,480
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,412
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,228
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,908
  7. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,268
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,120
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 731
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 720
  11. 2023Enabling Large Language Models to Generate Text with Citationscited by 588
  12. 2019ELI5: Long Form Question Answeringcited by 571
13 more
  1. 2024Knowledge Conflicts for LLMs: A Surveycited by 277
  2. 2020Evaluation of Text Generation: A Surveycited by 275
  3. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 234
  4. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  5. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  6. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  7. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  8. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  9. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  10. 2023Giraffe: Adventures in Expanding Context Lengths in LLMscited by 33
  11. 2024Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Textscited by 27
  12. 2024DebateQA: Evaluating Question Answering on Debatable Knowledgecited by 12
  13. 2024Groundedness in Retrieval-augmented Long-form Generation: An Empirical Studycited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 1; 2025: 23; 2026: 8.

32 papers.

Sort:

By area:

  1. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 243
  2. 2025Deep Research: A Survey of Autonomous Research Agentscited by 64
  3. 2025A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challengescited by 59
  4. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
  5. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  6. 2024Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Compositioncited by 16
  7. 2025LastingBench: Defend Benchmarks Against Knowledge Leakagecited by 14
  8. 2025TP-RAG: Benchmarking Retrieval-Augmented Large Language Model Agents for Spatiotemporal-Aware Travel Planningcited by 12
  9. 2025DeepSolution: Boosting Complex Engineering Solution Design via Tree-based Exploration and Bi-point Thinkingcited by 9
  10. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
  11. 2025FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMscited by 8
  12. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
20 more
  1. 2025SurveyGen: Quality-Aware Scientific Survey Generation with Large Language Modelscited by 7
  2. 2025Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAGcited by 7
  3. 2025RAVine: Reality-Aligned Evaluation for Agentic Searchcited by 6
  4. 2025RePro: Training Language Models to Faithfully Recycle the Web for Pretrainingcited by 6
  5. 2025Do LLMs Truly Understand When a Precedent Is Overruled?cited by 6
  6. 2026Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generationcited by 4
  7. 2025Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysiscited by 3
  8. 2025MHTS: Multi-Hop Tree Structure Framework for Generating Difficulty-Controllable QA Datasets for RAG Evaluationcited by 3
  9. 2025A Vision for Geo-Temporal Deep Research Systems: Towards Comprehensive, Transparent, and Reproducible Geo-Temporal Information Synthesiscited by 2
  10. 2025Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generationcited by 2
  11. 2025LongWeave: A Long-Form Generation Benchmark Bridging Real-World Relevance and Verifiabilitycited by 2
  12. 2026How Retrieved Context Shapes Internal Representations in RAGcited by 2
  13. 2025CMRAG: Co-modality-based visual document retrieval and question answeringcited by 1
  14. 2025An Open and Reproducible Deep Research Agent for Long-Form Question Answeringcited by 1
  15. 2026Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluationcited by 1
  16. 2026RAG-Stack: Co-Optimizing RAG Serving Performance and Qualitycited by 0
  17. 2026SurveyAgent-HKA: A multi-agent framework for scientific survey generation with LLMs and human knowledge augmentationcited by 0
  18. 2026Effects of Transcript Compression on LLM-based Medical Misinformation Detection in Japanese YouTube Videoscited by 0
  19. 2026DuplexSpeechBench-Document Grounding: Benchmarking Document Grounding and Hallucinations in Voice Agentscited by 0
  20. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models
  2. 2024In Defense of RAG in the Era of Long-Context Language Models
  3. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  4. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
  5. 2024CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity
  6. 2025PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation
  7. 2025Controlled Retrieval-augmented Context Evaluation for Long-form RAG
  8. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers