mlsanityData to 7 Oct 2026

RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering

arXiv 2407.13998 ↗First posted 19 Jul 2024cs.CL cs.AIcited by 43

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Question answering based on retrieval augmented generation (RAG-QA) is an important research topic in NLP and has a wide range of real-world applications. However, most existing datasets for this task are either constructed using a single source corpus or consist of short extractive answers, which fall short of evaluating large language model (LLM) based RAG-QA systems on cross-domain generalization. To address these limitations, we create Long-form RobustQA (LFRQA), a new dataset comprising human-written long-form answers that integrate short extractive answers from multiple documents into a single, coherent narrative, covering 26K queries and large corpora across seven different domains. We further propose RAG-QA Arena by directly comparing model-generated answers against LFRQA's answers using LLMs as evaluators. We show via extensive experiments that RAG-QA Arena and human judgments on answer quality are highly correlated. Moreover, only 41.3% of the most competitive LLM's answers are preferred to LFRQA's answers, demonstrating RAG-QA Arena as a challenging evaluation platform for future research.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 18 of its 43 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

18 of 43 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  9. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  10. 2022A Survey on In-context Learningcited by 717
  11. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  12. 2019ELI5: Long Form Question Answeringcited by 567
6 more
  1. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  2. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  3. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 253
  4. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  5. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  6. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 31; 2026: 9.

43 papers.

Sort:

By area:

  1. 2025Deep Research: A Survey of Autonomous Research Agentscited by 63
  2. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 60
  3. 2025Search Arena: Analyzing Search-Augmented LLMscited by 30
  4. 2025KET-RAG: A Cost-Efficient Multi-Granular Indexing Framework for Graph-RAGcited by 29
  5. 2025ArchRAG: Attributed Community-based Hierarchical Retrieval-Augmented Generationcited by 27
  6. 2025NodeRAG: Structuring Graph-based RAG with Heterogeneous Nodescited by 25
  7. 2025The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Modelscited by 24
  8. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 22
  9. 2025AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented Generationcited by 19
  10. 2025CiteEval: Principle-Driven Citation Evaluation for Source Attributioncited by 19
  11. 2025GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluationcited by 18
  12. 2025FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domaincited by 16
31 more
  1. 2025CoCoA: Confidence and Context-Aware Adaptive Decoding for Resolving Knowledge Conflicts in Large Language Modelscited by 9
  2. 2024Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coveragecited by 8
  3. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  4. 2025NyayaRAG: Realistic Legal Judgment Prediction with RAG under the Indian Common Law Systemcited by 5
  5. 2025RankArena: A Unified Platform for Evaluating Retrieval, Reranking and RAG with Human and LLM Feedbackcited by 5
  6. 2026Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revisioncited by 4
  7. 2025Balancing Truthfulness and Informativeness with Uncertainty-Aware Instruction Fine-Tuningcited by 3
  8. 2025Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysiscited by 3
  9. 2025Adapting Large Language Models for Multi-Domain Retrieval-Augmented-Generationcited by 3
  10. 2025LLM-as-a-qualitative-judge: automating error analysis in natural language generationcited by 3
  11. 2025MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question Answeringcited by 3
  12. 2025Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corporacited by 2
  13. 2025Reinforced Informativeness Optimization for Long-Form Retrieval-Augmented Generationcited by 2
  14. 2025How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAGcited by 2
  15. 2025Reasoning-Intensive Regressioncited by 2
  16. 2026DocArena: Turning Raw Documents into Controllable Training Environments for Document Search Agentscited by 2
  17. 2024A Survey of Query Optimization in Large Language Modelscited by 1
  18. 2025Reference-Aligned Retrieval-Augmented Question Answering over Heterogeneous Proprietary Documentscited by 1
  19. 2025LangProBe: a Language Programs Benchmarkcited by 1
  20. 2025GE-Chat: A Graph Enhanced RAG Framework for Evidential Response Generation of LLMscited by 1
  21. 2025Magic Mushroom: A Customizable Benchmark for Fine-grained Analysis of Retrieval Noise Erosion in RAG Systemscited by 1
  22. 2025Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generationcited by 1
  23. 2025RAIR: Retrieval-Augmented Iterative Refinement for Chinese Spelling Correctioncited by 0
  24. 2025The 3rd Place Solution of CCIR CUP 2025: A Framework for Retrieval-Augmented Generation in Multi-Turn Legal Conversationcited by 0
  25. 2026eTracer: Polarity-Aware Evidence Groundingcited by 0
  26. 2026SAGE: Steerable Agentic Data Generation for Deep Search with Execution Feedbackcited by 0
  27. 2026ArbGraph: Conflict-Aware Evidence Arbitration for Reliable Long-Form Retrieval-Augmented Generationcited by 0
  28. 2026ASTRA-QA: A Benchmark for Abstract Question Answering over Documentscited by 0
  29. 2026AILQA: Evaluating AI-Driven Legal Question Answering Systems for the Indian Legal Systemcited by 0
  30. 2026DOSA: A Tree-Guided, Self-Regressive Framework for Long Document Structure Analysiscited by 0
  31. 2026Checkpoints Are Not Enough: Trust Calibration in CoSLR, a Human-AI System for Systematic Literature Reviewscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Level
  2. 2025QuIM-RAG: Advancing Retrieval-Augmented Generation with Inverted Question Matching for Enhanced QA Performance
  3. 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering
  4. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  5. 2025On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systems
  6. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
  7. 2024CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systems
  8. 2026Augmenting Question Answering with A Hybrid RAG Approach