# FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models

arXiv: https://arxiv.org/abs/2402.14116 · HTML: https://mlsanity.com/paper/2402.14116/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 21 Feb 2024.

## Builds on (most cited first, 16 of 16)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://mlsanity.com/paper/2210.02406/) (2022)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [The Web as a Knowledge-base for Answering Complex Questions](https://mlsanity.com/paper/1803.06643/) (2018)
- [Text and Code Embeddings by Contrastive Pre-Training](https://mlsanity.com/paper/2201.10005/) (2022)
- [LooGLE: Can Long-Context Language Models Understand Long Contexts?](https://mlsanity.com/paper/2311.04939/) (2023)
- [ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding](https://mlsanity.com/paper/2305.14196/) (2023)
- [SCROLLS: Standardized CompaRison Over Long Language Sequences](https://mlsanity.com/paper/2201.03533/) (2022)
- [Compositional Questions Do Not Necessitate Multi-hop Reasoning](https://mlsanity.com/paper/1906.02900/) (2019)
- [GenDec: A robust generative Question-decomposition method for Multi-hop reasoning](https://mlsanity.com/paper/2402.11166/) (2024)
- [Kani: A Lightweight and Highly Hackable Framework for Building Language Model Applications](https://mlsanity.com/paper/2309.05542/) (2023)

## Built on by (most cited first, 29 of 29)

- [KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes](https://mlsanity.com/paper/2506.06541/) (2025)
- [CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning](https://mlsanity.com/paper/2503.13517/) (2025)
- [MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents](https://mlsanity.com/paper/2508.11133/) (2025)
- [Explicit v.s. Implicit Memory: Exploring Multi-hop Complex Reasoning Over Personalized Information](https://mlsanity.com/paper/2508.13250/) (2025)
- [General Agentic Planning Through Simulative Reasoning with World Models](https://mlsanity.com/paper/2507.23773/) (2025)
- [NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts](https://mlsanity.com/paper/2506.02000/) (2025)
- [Retrieval--Reasoning Processes for Multi-hop Question Answering: A Four-Axis Design Framework and Empirical Trends](https://mlsanity.com/paper/2601.00536/) (2026)
- [WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflection](https://mlsanity.com/paper/2510.18798/) (2025)
- [CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning](https://mlsanity.com/paper/2601.14952/) (2026)
- [Tracking the Limits of Knowledge Propagation: How LLMs Fail at Multi-Step Reasoning with Conflicting Knowledge](https://mlsanity.com/paper/2601.15495/) (2026)
- [TACT: Advancing Complex Aggregative Reasoning with Information Extraction Tools](https://mlsanity.com/paper/2406.03618/) (2024)
- [Cognitive Load Limits in Large Language Models: Benchmarking Multi-Hop Reasoning](https://mlsanity.com/paper/2509.19517/) (2025)
- [Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics](https://mlsanity.com/paper/2510.05137/) (2025)
- [PEEK: Context Map as an Orientation Cache for Long-Context LLM Agents](https://mlsanity.com/paper/2605.19932/) (2026)
- [OntoRAG: Enhancing Question-Answering through Automated Ontology Derivation from Unstructured Knowledge Bases](https://mlsanity.com/paper/2506.00664/) (2025)
- [MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance](https://mlsanity.com/paper/2506.14927/) (2025)
- [Do Reasoning LLMs Refuse What They Infer in Long Contexts?](https://mlsanity.com/paper/2602.08874/) (2026)
- [BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidence](https://mlsanity.com/paper/2603.07931/) (2026)
- [Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspective](https://mlsanity.com/paper/2510.08800/) (2025)
- [RVR: Retrieve-Verify-Retrieve for Comprehensive Question Answering](https://mlsanity.com/paper/2602.18425/) (2026)
- [When Cloud Agents Meet Device Agents: Lessons from Hybrid Multi-Agent Systems](https://mlsanity.com/paper/2605.30102/) (2026)
- [Query Generation Pipeline with Enhanced Answerability Assessment for Financial Information Retrieval](https://mlsanity.com/paper/2511.05000/) (2025)
- [Supporting software engineering tasks with agentic AI: Demonstration on document retrieval and test scenario generation](https://mlsanity.com/paper/2602.04726/) (2026)
- [π2: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2604.05114/) (2026)
- [BANDMAS: Causality-Inspired Semantic Packet Scheduling for Bandwidth-Efficient Multi-Agent Collaboration](https://mlsanity.com/paper/2608.00458/) (2026)
- [SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information](https://mlsanity.com/paper/2608.10692/) (2026)
- [Evidence Blindness in Direct Corpus Interaction: Persistent Navigation with AtlasNav](https://mlsanity.com/paper/2608.24764/) (2026)
- [Receiver-Conditioned Latent Communication gives 94% CacheBack](https://mlsanity.com/paper/2609.32046/) (2026)
- [Overlap, Unique and Conflict: Can LLMs Extract What They Can Recognize?](https://mlsanity.com/paper/2609.38799/) (2026)

## Abstract (arXiv)

One type of question that is commonly found in day-to-day scenarios is ``fan-out'' questions, complex multi-hop, multi-document reasoning questions that require finding information about a large number of entities. However, there exist few resources to evaluate this type of question-answering capability among large language models. To evaluate complex reasoning in LLMs more fully, we present FanOutQA, a high-quality dataset of fan-out question-answer pairs and human-annotated decompositions with English Wikipedia as the knowledge base. We formulate three benchmark settings across our dataset and benchmark 7 LLMs, including GPT-4, LLaMA 2, Claude-2.1, and Mixtral-8x7B, finding that contemporary models still have room to improve reasoning over inter-document dependencies in a long context. We provide our dataset and open-source tools to run models to encourage evaluation at https://fanoutqa.com