# DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections

arXiv: https://arxiv.org/abs/2508.15851 · HTML: https://mlsanity.com/paper/2508.15851/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 31 of its 38 references that are in the map.
- First posted on arXiv 20 Aug 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 30 of 31)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data](https://mlsanity.com/paper/2004.07347/) (2020)
- [Constructing Datasets for Multi-hop Reading Comprehension Across Documents](https://mlsanity.com/paper/1710.06481/) (2017)
- [SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images](https://mlsanity.com/paper/2301.04883/) (2023)
- [BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions](https://mlsanity.com/paper/2308.09936/) (2023)
- [Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA](https://mlsanity.com/paper/2406.17419/) (2024)
- [SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers](https://mlsanity.com/paper/2407.09413/) (2024)
- [ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogram](https://mlsanity.com/paper/2306.15681/) (2023)
- [Multi-hop Question Answering via Reasoning Chains](https://mlsanity.com/paper/1910.02610/) (2019)
- [VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.10704/) (2024)
- [MultiTabQA: Generating Tabular Answers for Multi-Table Question Answering](https://mlsanity.com/paper/2305.12820/) (2023)
- [M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2411.04075/) (2024)
- [Mitigating Lost-in-Retrieval Problems in Retrieval Augmented Multi-Hop Question Answering](https://mlsanity.com/paper/2502.14245/) (2025)
- [FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining](https://mlsanity.com/paper/2109.07323/) (2021)
- [LIQUID: A Framework for List Question Answering Dataset Generation](https://mlsanity.com/paper/2302.01691/) (2023)
- [MEBench: Benchmarking Large Language Models for Cross-Document Multi-Entity Question Answering](https://mlsanity.com/paper/2502.18993/) (2025)
- [Exploiting Reasoning Chains for Multi-hop Science Question Answering](https://mlsanity.com/paper/2109.02905/) (2021)
- [Sequence tagging for biomedical extractive question answering](https://mlsanity.com/paper/2104.07535/) (2021)
- [BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain](https://mlsanity.com/paper/2505.22240/) (2025)
- [NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts](https://mlsanity.com/paper/2506.02000/) (2025)

## Built on by (most cited first, 8 of 8)

- [MLDocRAG: Multimodal Long-Context Document Retrieval Augmented Generation](https://mlsanity.com/paper/2602.10271/) (2026)
- [PaperMind: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMs](https://mlsanity.com/paper/2604.21304/) (2026)
- [When Iterative RAG Beats Ideal Evidence: A Diagnostic Study in Scientific Multi-hop Question Answering](https://mlsanity.com/paper/2601.19827/) (2026)
- [Automatic Inter-document Multi-hop Scientific QA Generation](https://mlsanity.com/paper/2603.14257/) (2026)
- [DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents](https://mlsanity.com/paper/2609.02059/) (2026)
- [Automating Multi-Hop RAG Evaluation via TRIAD: From Context Extraction to Validated Dataset Generation](https://mlsanity.com/paper/2608.21558/) (2026)
- [MechReason: Benchmarking Multi-Image Multi-Hop Reasoning in Mechanical Engineering](https://mlsanity.com/paper/2609.16012/) (2026)
- [TopoGraphRAG-Bench: Evaluating Multimodal GraphRAG on Layout-Grounded Evidence Reasoning](https://mlsanity.com/paper/2610.09360/) (2026)

## Abstract (arXiv)

Despite rapid progress in large language models (LLMs), current QA benchmarks still overlook the core challenge of real-world scientific information seeking: synthesizing multimodal evidence scattered across multiple documents and structural formats. Existing QAs remain narrow in scope, relying on unimodal text and short-span reasoning that fail to capture the complexity of real information-seeking. We introduce DocHop-QA, a benchmark of 11,379 instances for evaluating multimodal, multi-document, multi-hop scientific QA. Built from publicly available PubMed articles, DocHop-QA incorporates textual passages, tables, and layout cues, enabling cross-document inference without explicit hyperlinks. To scale realistic QA construction, we develop an LLM-driven generation pipeline grounded in 11 scientific reasoning concepts, producing diverse and coherent question-answer pairs. To highlight the utility and versatility of the dataset, we propose a task-driven evaluation framework spanning four settings, including generative answering, multimodal evidence integration and structured index prediction. Experiments show that current models struggle with DocHop-QA's long-context, multi-evidence demands, establishing it as a rigorous testbed for advancing next-generation scientific QA systems.