# Constructing Datasets for Multi-hop Reading Comprehension Across Documents

arXiv: https://arxiv.org/abs/1710.06481 · HTML: https://mlsanity.com/paper/1710.06481/

## Key facts

- Cited by 190 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Oct 2017.

## Built on by (most cited first, 30 of 190)

- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [Making Retrieval-Augmented Language Models Robust to Irrelevant Context](https://mlsanity.com/paper/2310.01558/) (2023)
- [Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning](https://mlsanity.com/paper/2205.09712/) (2022)
- [QuALITY: Question Answering with Long Input Texts, Yes!](https://mlsanity.com/paper/2112.08608/) (2021)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023)
- [MultiModalQA: Complex Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2104.06039/) (2021)
- [Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering](https://mlsanity.com/paper/2101.00774/) (2021)
- [Do Large Language Models Latently Perform Multi-Hop Reasoning?](https://mlsanity.com/paper/2402.16837/) (2024)
- [Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.12941/) (2024)
- [CMB: A Comprehensive Medical Benchmark in Chinese](https://mlsanity.com/paper/2308.08833/) (2023)
- [WebQA: Multihop and Multimodal QA](https://mlsanity.com/paper/2109.00590/) (2021)
- [Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories](https://mlsanity.com/paper/2306.09224/) (2023)
- [Answering Questions by Meta-Reasoning over Multiple Chains of Thought](https://mlsanity.com/paper/2304.13007/) (2023)
- [Towards Mitigating Hallucination in Large Language Models via Self-Reflection](https://mlsanity.com/paper/2310.06271/) (2023)
- [Exploring the Benefits of Training Expert Language Models over Instruction Tuning](https://mlsanity.com/paper/2302.03202/) (2023)
- [A Survey of Multimodal Large Language Model from A Data-centric Perspective](https://mlsanity.com/paper/2405.16640/) (2024)
- [SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers](https://mlsanity.com/paper/2407.09413/) (2024)
- [StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts](https://mlsanity.com/paper/2204.08292/) (2022)
- [Differentially Private Optimization on Large Model at Small Cost](https://mlsanity.com/paper/2210.00038/) (2022)
- [A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains](https://mlsanity.com/paper/2402.00559/) (2024)
- [Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval](https://mlsanity.com/paper/2101.00436/) (2021)
- [What to Pre-Train on? Efficient Intermediate Task Selection](https://mlsanity.com/paper/2104.08247/) (2021)
- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases](https://mlsanity.com/paper/2404.13207/) (2024)
- [AR-LSAT: Investigating Analytical Reasoning of Text](https://mlsanity.com/paper/2104.06598/) (2021)
- [Biomedical Question Answering: A Survey of Approaches and Challenges](https://mlsanity.com/paper/2102.05281/) (2021)
- [Huatuo-26M, a Large-scale Chinese Medical QA Dataset](https://mlsanity.com/paper/2305.01526/) (2023)

## Abstract (arXiv)

Most Reading Comprehension methods limit themselves to queries which can be answered using a single sentence, paragraph, or document. Enabling models to combine disjoint pieces of textual evidence would extend the scope of machine comprehension methods, but currently there exist no resources to train and test this capability. We propose a novel task to encourage the development of models for text understanding across multiple documents and to investigate the limits of existing methods. In our task, a model learns to seek and combine evidence - effectively performing multi-hop (alias multi-step) inference. We devise a methodology to produce datasets for this task, given a collection of query-answer pairs and thematically linked documents. Two datasets from different domains are induced, and we identify potential pitfalls and devise circumvention strategies. We evaluate two previously proposed competitive models and find that one can integrate information across documents. However, both models struggle to select relevant information, as providing documents guaranteed to be relevant greatly improves their performance. While the models outperform several strong baselines, their best accuracy reaches 42.9% compared to human performance at 74.0% - leaving ample room for improvement.