mlsanityData to 13 Oct 2026

Understanding Dataset Design Choices for Multi-hop Reasoning

arXiv 1904.12106 ↗First posted 27 Apr 2019cs.CL cs.AIcited by 50

Not on the map. Why?

Abstract (arXiv)

Learning multi-hop reasoning has been a key challenge for reading comprehension models, leading to the design of datasets that explicitly focus on it. Ideally, a model should not be able to perform well on a multi-hop question answering task without doing multi-hop reasoning. In this paper, we investigate two recently proposed datasets, WikiHop and HotpotQA. First, we explore sentence-factored models for these tasks; by design, these models cannot do multi-hop reasoning, but they are still able to solve a large number of examples in both datasets. Furthermore, we find spurious correlations in the unmasked version of WikiHop, which make it easy to achieve high performance considering only the questions and answers. Finally, we investigate one key difference between these datasets, namely span-based vs. multiple-choice formulations of the QA task. Multiple-choice versions of both datasets can be easily gamed, and two models we examine only marginally exceed a baseline in this setting. Overall, while these datasets are useful testbeds, high-performing models may not be learning as much multi-hop reasoning as previously thought.

Key facts. Cited by 50 papers in the map (data through 13 Oct 2026). First posted on arXiv 27 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 12; 2022: 14; 2023: 5; 2024: 5; 2025: 8; 2026: 6.

50 papers.

Sort:

By area:

  1. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 902
  2. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  3. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 151
  4. 2021Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrievalcited by 63
  5. 2023Natural Language Reasoning, A Surveycited by 61
  6. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 52
  7. 2021Can NLI Models Verify QA Systems' Predictions?cited by 46
  8. 2021Joint Passage Ranking for Diverse Multi-Answer Retrievalcited by 43
  9. 2022QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphscited by 42
  10. 2024MoreHopQA: More Than Multi-hop Reasoningcited by 37
  11. 2021Challenges in Generalization in Open Domain Question Answeringcited by 30
  12. 2021Flexible Generation of Natural Language Deductionscited by 28
38 more
  1. 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 25
  2. 2021Rissanen Data Analysis: Examining Dataset Characteristics via Description Lengthcited by 22
  3. 2022RobustLR: Evaluating Robustness to Logical Perturbation in Deductive Reasoningcited by 22
  4. 2022Reasoning over Public and Private Data in Retrieval-Based Systemscited by 21
  5. 2021Connecting Attributions and QA Model Behavior on Realistic Counterfactualscited by 20
  6. 2021Can Explanations Be Useful for Calibrating Black Box Models?cited by 20
  7. 2021Uncertainty Calibration for Ensemble-Based Debiasing Methodscited by 16
  8. 2022TASA: Deceiving Question Answering Models by Twin Answer Sentences Attackcited by 15
  9. 2025FactCG: Enhancing Fact Checkers with Graph-Based Multi-Hop Datacited by 14
  10. 2021Robustifying Multi-hop QA through Pseudo-Evidentiality Trainingcited by 13
  11. 2022Machine Reading, Fast and Slow: When Do Models "Understand" Language?cited by 13
  12. 2022What Makes Reading Comprehension Questions Difficult?cited by 11
  13. 2023What Else Do I Need to Know? The Effect of Background Information on Users' Reliance on QA Systemscited by 9
  14. 2022Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contextscited by 7
  15. 2022Understanding and Improving Zero-shot Multi-hop Reasoning in Generative Question Answeringcited by 7
  16. 2022Assessing Out-of-Domain Language Model Performance from Few Examplescited by 6
  17. 2023HOP, UNION, GENERATE: Explainable Multi-hop Reasoning without Rationale Supervisioncited by 5
  18. 2025GlobalRAG: Enhancing Global Reasoning in Multi-hop Question Answering via Reinforcement Learningcited by 5
  19. 2022A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehensioncited by 4
  20. 2022ReasonChainQA: Text-based Complex Question Answering with Explainable Evidence Chainscited by 4
  21. 2025Replace, Don't Expand: Mitigating Context Dilution in Multi-Hop RAG via Fixed-Budget Evidence Assemblycited by 3
  22. 2021Model Agnostic Answer Reranking System for Adversarial Question Answeringcited by 2
  23. 2023Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answeringcited by 2
  24. 2024Geometric Analysis of Reasoning Trajectories: A Phase Space Approach to Understanding Valid and Invalid Multi-Hop Reasoning in LLMscited by 2
  25. 2022How Well Do Multi-hop Reading Comprehension Models Understand Date Information?cited by 1
  26. 2024ACCORD: Closing the Commonsense Measurability Gapcited by 1
  27. 2025π-CoT: Prolog-Initialized Chain-of-Thought Prompting for Multi-Hop Question-Answeringcited by 1
  28. 2025Leveraging Spreading Activation for Improved Document Retrieval in Knowledge-Graph-Based RAG Systemscited by 1
  29. 2026RUBEN: Rule-Based Explanations for Retrieval-Augmented LLM Systemscited by 1
  30. 2026Helicase: Uncertainty-Guided Supply Chain Knowledge Graph Construction with Autonomous Multi-Agent LLMscited by 1
  31. 2022MRCLens: an MRC Dataset Bias Detection Toolkitcited by 0
  32. 2024Evaluating Long Range Dependency Handling in Code Generation LLMscited by 0
  33. 2025Rule-Based Explanations for Retrieval-Augmented LLM Systemscited by 0
  34. 2025How Prompts Move Language Model Behavior: Frames, Salience, and Construal as Semantic Controlcited by 0
  35. 2026GRASP: Graph Agentic Search over Propositions for Multi-hop Question Answeringcited by 0
  36. 2026WildTrace: Benchmarking Natural Evidence Trails in Long-Context Reasoningcited by 0
  37. 2026NS-ST-GraphRAG: Neuro-Symbolic Spatio-Temporal GraphRAG for Literary Knowledge Processingcited by 0
  38. 2026REALHOP: Rethinking Multi-Hop Reasoning Evaluation via Behavioral Auditingcited by 0