mlsanityData to 14 Oct 2026

Evaluation of retrieval-based QA on QUEST-LOFT

arXiv 2511.06125 ↗First posted 8 Nov 2025cs.CL cs.AI cs.IRcited by 0

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Despite the popularity of retrieval-augmented generation (RAG) as a solution for grounded QA in both academia and industry, current RAG methods struggle with questions where the necessary information is distributed across many documents or where retrieval needs to be combined with complex reasoning. Recently, the LOFT study has shown that this limitation also applies to approaches based on long-context language models, with the QUEST benchmark exhibiting particularly large headroom. In this paper, we provide an in-depth analysis of the factors contributing to the poor performance on QUEST-LOFT, publish updated numbers based on a thorough human evaluation, and demonstrate that RAG can be optimized to significantly outperform long-context approaches when combined with a structured output format containing reasoning and evidence, optionally followed by answer re-verification.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 22 of its 30 references that are in the map. First posted on arXiv 8 Nov 2025.

Builds on

22 of 30 references.

References · 22

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  6. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  7. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  8. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  9. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  10. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  11. 2023Language Models can Solve Computer Taskscited by 372
  12. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 324
10 more
  1. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  2. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 159
  3. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 156
  4. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 119
  5. 2022Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Modelscited by 112
  6. 2024Self-Discover: Large Language Models Self-Compose Reasoning Structurescited by 110
  7. 2021Joint Passage Ranking for Diverse Multi-Answer Retrievalcited by 43
  8. 2023QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operationscited by 31
  9. 2024Atomic Self-Consistency for Better Long Form Generationscited by 10
  10. 2024SetBERT: Enhancing Retrieval Performance for Boolean Logic and Set Operation Queriescited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models
  2. 2024Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective
  3. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  4. 2025On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systems
  5. 2025T2-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation
  6. 2024Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems
  7. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  8. 2026Augmenting Question Answering with A Hybrid RAG Approach