# Long2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall

arXiv: https://arxiv.org/abs/2410.23000 · HTML: https://mlsanity.com/paper/2410.23000/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 28 of 28)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [Knowledge Conflicts for LLMs: A Survey](https://mlsanity.com/paper/2403.08319/) (2024)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2311.09476/) (2023)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [ASQA: Factoid Questions Meet Long-Form Answers](https://mlsanity.com/paper/2204.06092/) (2022)
- [ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding](https://mlsanity.com/paper/2305.14196/) (2023)
- [Hurdles to Progress in Long-form Question Answering](https://mlsanity.com/paper/2103.06332/) (2021)
- [Rethinking with Retrieval: Faithful Large Language Model Inference](https://mlsanity.com/paper/2301.00303/) (2022)
- [A Critical Evaluation of Evaluations for Long-form Question Answering](https://mlsanity.com/paper/2305.18201/) (2023)
- [RankGen: Improving Text Generation with Large Ranking Models](https://mlsanity.com/paper/2205.09726/) (2022)
- [The Chronicles of RAG: The Retriever, the Chunk and the Generator](https://mlsanity.com/paper/2401.07883/) (2024)
- [Giraffe: Adventures in Expanding Context Lengths in LLMs](https://mlsanity.com/paper/2308.10882/) (2023)
- [Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Texts](https://mlsanity.com/paper/2402.07625/) (2024)
- [DebateQA: Evaluating Question Answering on Debatable Knowledge](https://mlsanity.com/paper/2408.01419/) (2024)
- [Groundedness in Retrieval-augmented Long-form Generation: An Empirical Study](https://mlsanity.com/paper/2404.07060/) (2024)

## Built on by (most cited first, 30 of 32)

- [Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions](https://mlsanity.com/paper/2507.05257/) (2025)
- [Deep Research: A Survey of Autonomous Research Agents](https://mlsanity.com/paper/2508.12752/) (2025)
- [A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges](https://mlsanity.com/paper/2508.05668/) (2025)
- [LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routing](https://mlsanity.com/paper/2502.09977/) (2025)
- [Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.14891/) (2025)
- [Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Composition](https://mlsanity.com/paper/2412.13612/) (2024)
- [LastingBench: Defend Benchmarks Against Knowledge Leakage](https://mlsanity.com/paper/2506.21614/) (2025)
- [TP-RAG: Benchmarking Retrieval-Augmented Large Language Model Agents for Spatiotemporal-Aware Travel Planning](https://mlsanity.com/paper/2504.08694/) (2025)
- [DeepSolution: Boosting Complex Engineering Solution Design via Tree-based Exploration and Bi-point Thinking](https://mlsanity.com/paper/2502.20730/) (2025)
- [Memory in Large Language Models: Mechanisms, Evaluation and Evolution](https://mlsanity.com/paper/2509.18868/) (2025)
- [FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMs](https://mlsanity.com/paper/2510.12839/) (2025)
- [Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets](https://mlsanity.com/paper/2504.20119/) (2025)
- [SurveyGen: Quality-Aware Scientific Survey Generation with Large Language Models](https://mlsanity.com/paper/2508.17647/) (2025)
- [Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG](https://mlsanity.com/paper/2509.13930/) (2025)
- [RAVine: Reality-Aligned Evaluation for Agentic Search](https://mlsanity.com/paper/2507.16725/) (2025)
- [RePro: Training Language Models to Faithfully Recycle the Web for Pretraining](https://mlsanity.com/paper/2510.10681/) (2025)
- [Do LLMs Truly Understand When a Precedent Is Overruled?](https://mlsanity.com/paper/2510.20941/) (2025)
- [Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generation](https://mlsanity.com/paper/2604.03141/) (2026)
- [Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysis](https://mlsanity.com/paper/2503.16575/) (2025)
- [MHTS: Multi-Hop Tree Structure Framework for Generating Difficulty-Controllable QA Datasets for RAG Evaluation](https://mlsanity.com/paper/2504.08756/) (2025)
- [A Vision for Geo-Temporal Deep Research Systems: Towards Comprehensive, Transparent, and Reproducible Geo-Temporal Information Synthesis](https://mlsanity.com/paper/2506.14345/) (2025)
- [Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation](https://mlsanity.com/paper/2510.07926/) (2025)
- [LongWeave: A Long-Form Generation Benchmark Bridging Real-World Relevance and Verifiability](https://mlsanity.com/paper/2510.24345/) (2025)
- [How Retrieved Context Shapes Internal Representations in RAG](https://mlsanity.com/paper/2602.20091/) (2026)
- [Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation](https://mlsanity.com/paper/2604.20763/) (2026)
- [CMRAG: Co-modality-based visual document retrieval and question answering](https://mlsanity.com/paper/2509.02123/) (2025)
- [An Open and Reproducible Deep Research Agent for Long-Form Question Answering](https://mlsanity.com/paper/2512.13059/) (2025)
- [RAG-Stack: Co-Optimizing RAG Serving Performance and Quality](https://mlsanity.com/paper/2608.03487/) (2026)
- [SurveyAgent-HKA: A multi-agent framework for scientific survey generation with LLMs and human knowledge augmentation](https://mlsanity.com/paper/2609.05938/) (2026)
- [Effects of Transcript Compression on LLM-based Medical Misinformation Detection in Japanese YouTube Videos](https://mlsanity.com/paper/2609.30882/) (2026)

## Abstract (arXiv)

Retrieval-augmented generation (RAG) is a promising approach to address the limitations of fixed knowledge in large language models (LLMs). However, current benchmarks for evaluating RAG systems suffer from two key deficiencies: (1) they fail to adequately measure LLMs' capability in handling long-context retrieval due to a lack of datasets that reflect the characteristics of retrieved documents, and (2) they lack a comprehensive evaluation method for assessing LLMs' ability to generate long-form responses that effectively exploits retrieved information. To address these shortcomings, we introduce the Long2RAG benchmark and the Key Point Recall (KPR) metric. Long2RAG comprises 280 questions spanning 10 domains and across 8 question categories, each associated with 5 retrieved documents with an average length of 2,444 words. KPR evaluates the extent to which LLMs incorporate key points extracted from the retrieved documents into their generated responses, providing a more nuanced assessment of their ability to exploit retrieved information.