mlsanityData to 14 Oct 2026

Constructing Datasets for Multi-hop Reading Comprehension Across Documents

arXiv 1710.06481 ↗First posted 17 Oct 2017cs.CL cs.AIcited by 190

Not on the map. Why?

Cited by 38 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Most Reading Comprehension methods limit themselves to queries which can be answered using a single sentence, paragraph, or document. Enabling models to combine disjoint pieces of textual evidence would extend the scope of machine comprehension methods, but currently there exist no resources to train and test this capability. We propose a novel task to encourage the development of models for text understanding across multiple documents and to investigate the limits of existing methods. In our task, a model learns to seek and combine evidence - effectively performing multi-hop (alias multi-step) inference. We devise a methodology to produce datasets for this task, given a collection of query-answer pairs and thematically linked documents. Two datasets from different domains are induced, and we identify potential pitfalls and devise circumvention strategies. We evaluate two previously proposed competitive models and find that one can integrate information across documents. However, both models struggle to select relevant information, as providing documents guaranteed to be relevant greatly improves their performance. While the models outperform several strong baselines, their best accuracy reaches 42.9% compared to human performance at 74.0% - leaving ample room for improvement.

Key facts. Cited by 190 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 49; 2022: 44; 2023: 26; 2024: 26; 2025: 23; 2026: 22.

190 papers.

Sort:

By area:

  1. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  2. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  3. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  4. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 408
  5. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  6. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 334
  7. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  8. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  9. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 179
  10. 2021Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answeringcited by 178
  11. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 169
  12. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 162
38 more
  1. 2023CMB: A Comprehensive Medical Benchmark in Chinesecited by 115
  2. 2021WebQA: Multihop and Multimodal QAcited by 109
  3. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 107
  4. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  5. 2023Towards Mitigating Hallucination in Large Language Models via Self-Reflectioncited by 81
  6. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  7. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  8. 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
  9. 2022StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Textscited by 68
  10. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  11. 2024A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chainscited by 65
  12. 2021Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrievalcited by 63
  13. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 63
  14. 2023Natural Language Reasoning, A Surveycited by 61
  15. 2024STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Basescited by 55
  16. 2021AR-LSAT: Investigating Analytical Reasoning of Textcited by 51
  17. 2021Biomedical Question Answering: A Survey of Approaches and Challengescited by 46
  18. 2023Huatuo-26M, a Large-scale Chinese Medical QA Datasetcited by 46
  19. 2023Can Language Models be Instructed to Protect Personal Information?cited by 46
  20. 2021From LSAT: The Progress and Challenges of Complex Reasoningcited by 43
  21. 2022A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methodscited by 42
  22. 2022QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphscited by 42
  23. 2023Pre-Training to Learn in Contextcited by 42
  24. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inferencecited by 40
  25. 2024MoreHopQA: More Than Multi-hop Reasoningcited by 37
  26. 2022BigBIO: A Framework for Data-Centric Biomedical Natural Language Processingcited by 36
  27. 2021Decomposing Complex Questions Makes Multi-Hop QA Easier and More Interpretablecited by 35
  28. 2025DrDiff: Dynamic Routing Diffusion with Hierarchical Attention for Breaking the Efficiency-Quality Trade-offcited by 34
  29. 2021ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answerscited by 29
  30. 2021DAGN: Discourse-Aware Graph Network for Logical Reasoningcited by 28
  31. 2021Flexible Generation of Natural Language Deductionscited by 28
  32. 2023Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answeringcited by 28
  33. 2023CRAB: Assessing the Strength of Causal Relationships Between Real-world Eventscited by 28
  34. 2022SKILL: Structured Knowledge Infusion for Large Language Modelscited by 27
  35. 2021English Machine Reading Comprehension Datasets: A Surveycited by 26
  36. 2024Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generationcited by 25
  37. 2022Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Modelscited by 24
  38. 2022Modern Question Answering Datasets and Benchmarks: A Surveycited by 23

50 most cited of 190 All 190 (JSON)