mlsanityData to 14 Oct 2026

FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models

arXiv 2402.14116 ↗First posted 21 Feb 2024cs.CL cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

One type of question that is commonly found in day-to-day scenarios is ``fan-out'' questions, complex multi-hop, multi-document reasoning questions that require finding information about a large number of entities. However, there exist few resources to evaluate this type of question-answering capability among large language models. To evaluate complex reasoning in LLMs more fully, we present FanOutQA, a high-quality dataset of fan-out question-answer pairs and human-annotated decompositions with English Wikipedia as the knowledge base. We formulate three benchmark settings across our dataset and benchmark 7 LLMs, including GPT-4, LLaMA 2, Claude-2.1, and Mixtral-8x7B, finding that contemporary models still have room to improve reasoning over inter-document dependencies in a long context. We provide our dataset and open-source tools to run models to encourage evaluation at https://fanoutqa.com

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 16 of its 38 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

16 of 38 references.

References · 16

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  7. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  8. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  9. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 410
  10. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  11. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  12. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
4 more
  1. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  2. 2019Compositional Questions Do Not Necessitate Multi-hop Reasoningcited by 88
  3. 2024GenDec: A robust generative Question-decomposition method for Multi-hop reasoningcited by 13
  4. 2023Kani: A Lightweight and Highly Hackable Framework for Building Language Model Applicationscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 15.

29 papers.

Sort:

By area:

  1. 2025KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakescited by 33
  2. 2025CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoningcited by 28
  3. 2025MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documentscited by 21
  4. 2025Explicit v.s. Implicit Memory: Exploring Multi-hop Complex Reasoning Over Personalized Informationcited by 17
  5. 2025General Agentic Planning Through Simulative Reasoning with World Modelscited by 13
  6. 2025NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contextscited by 8
  7. 2026Retrieval--Reasoning Processes for Multi-hop Question Answering: A Four-Axis Design Framework and Empirical Trendscited by 8
  8. 2025WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflectioncited by 7
  9. 2026CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoningcited by 7
  10. 2026Tracking the Limits of Knowledge Propagation: How LLMs Fail at Multi-Step Reasoning with Conflicting Knowledgecited by 7
  11. 2024TACT: Advancing Complex Aggregative Reasoning with Information Extraction Toolscited by 5
  12. 2025Cognitive Load Limits in Large Language Models: Benchmarking Multi-Hop Reasoningcited by 5
17 more
  1. 2025Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metricscited by 4
  2. 2026PEEK: Context Map as an Orientation Cache for Long-Context LLM Agentscited by 4
  3. 2025OntoRAG: Enhancing Question-Answering through Automated Ontology Derivation from Unstructured Knowledge Basescited by 3
  4. 2025MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidancecited by 3
  5. 2026Do Reasoning LLMs Refuse What They Infer in Long Contexts?cited by 3
  6. 2026BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidencecited by 3
  7. 2025Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspectivecited by 1
  8. 2026RVR: Retrieve-Verify-Retrieve for Comprehensive Question Answeringcited by 1
  9. 2026When Cloud Agents Meet Device Agents: Lessons from Hybrid Multi-Agent Systemscited by 1
  10. 2025Query Generation Pipeline with Enhanced Answerability Assessment for Financial Information Retrievalcited by 0
  11. 2026Supporting software engineering tasks with agentic AI: Demonstration on document retrieval and test scenario generationcited by 0
  12. 2026π2: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Modelscited by 0
  13. 2026BANDMAS: Causality-Inspired Semantic Packet Scheduling for Bandwidth-Efficient Multi-Agent Collaborationcited by 0
  14. 2026SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Informationcited by 0
  15. 2026Evidence Blindness in Direct Corpus Interaction: Persistent Navigation with AtlasNavcited by 0
  16. 2026Receiver-Conditioned Latent Communication gives 94% CacheBackcited by 0
  17. 2026Overlap, Unique and Conflict: Can LLMs Extract What They Can Recognize?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge
  2. 2025MEBench: Benchmarking Large Language Models for Cross-Document Multi-Entity Question Answering
  3. 2026LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake
  4. 2025HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questions
  5. 2025KG-o1: Enhancing Multi-hop Question Answering in Large Language Models via Knowledge Graph Integration
  6. 2025DEEPAMBIGQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness
  7. 2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens
  8. 2023Multimodal Multi-Hop Question Answering Through a Conversation Between Tools and Efficiently Finetuned Large Language Models