VISPATH: Visual-Intent-Guided Path Reasoning for Multimodal Knowledge Graph Question Answering
Not on the map. Why?
Abstract (arXiv)
Knowledge graph question answering (KGQA) enables models to answer natural-language questions through structured graph reasoning and has achieved substantial progress across many benchmarks and applications. Recently, multimodal KGQA (MM-KGQA) has attracted increasing attention because many questions require jointly using multimodal inputs and KG evidence. However, existing MM-KGQA methods typically use multimodal information only for starting entity grounding or evidence retrieval, after which multi-hop reasoning degenerates into text-only graph search. As a result, they cannot exploit multimodal cues that become important at intermediate hops. To address this limitation, we propose VISPATH, a visual-intent-guided path reasoning framework for MM-KGQA. VISPATH first identifies a reliable starting entity by combining multimodal grounding with graph-structural cues. It then performs intent-guided path discovery by recomputing hop-specific multimodal intent from the input, question, and current partial paths, so that each expansion is guided by the current reasoning state. The discovered paths are further refined through reasoning-chain pruning, which evaluates candidate paths as complete evidence chains based on their consistency with the question, reasoning sketch, and hop-specific intent. Finally, VISPATH checks whether the selected evidence is sufficient for answer generation. We further construct VISPATH-Bench, a benchmark for evaluating multimodal multi-hop reasoning over KGs, covering questions that require two to four hops over KG paths. Extensive experiments on VISPATH-Bench and three additional multimodal QA benchmarks show that VISPATH consistently outperforms strong baselines. Notably, with GPT-4o as the backbone, VISPATH surpasses GPT-5.4 on VISPATH-Bench, achieving a 10.6% relative improvement in average accuracy and a 13.1% improvement at 2-hop reasoning.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 58 references that are in the map. First posted on arXiv 4 Aug 2026.
Builds on
39 of 58 references.
References · 39
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024GPT-4o System Cardcited by 4,467
- 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
- 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
- 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
- 2020Knowledge Graphscited by 619
- 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
- 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 423
- 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
- 2023Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoningcited by 326
27 more
- 2024LightRAG: Simple and Fast Retrieval-Augmented Generationcited by 310
- 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
- 2024G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answeringcited by 217
- 2019PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Textcited by 182
- 2018Open Domain Question Answering Using Early Fusion of Knowledge Bases and Textcited by 167
- 2023Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sourcescited by 121
- 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 120
- 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
- 2019MMKG: Multi-Modal Knowledge Graphscited by 104
- 2021Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signalscited by 100
- 2021RnG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answeringcited by 86
- 2024Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphscited by 77
- 2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
- 2023MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Modelscited by 73
- 2016Image-embodied Knowledge Representation Learningcited by 68
- 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agentcited by 65
- 2024Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generationcited by 63
- 2021TransferNet: An Effective and Transparent Framework for Multi-hop Question Answering over Relation Graphcited by 60
- 2024Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoningcited by 44
- 2020VisualSem: A High-quality Knowledge Graph for Vision and Languagecited by 30
- 2023UKnow: A Unified Knowledge Protocol with Multimodal Knowledge Graph Datasets for Reasoning and Vision-Language Pre-Trainingcited by 14
- 2025mKG-RAG: Leveraging Multimodal Knowledge Graphs in Retrieval-Augmented Generation for Knowledge-intensive VQAcited by 10
- 2025ReasonVQA: A Multi-hop Reasoning Benchmark with Structural Knowledge for Visual Question Answeringcited by 7
- 2023AspectMMKG: A Multi-modal Knowledge Graph with Aspect-aware Entitiescited by 5
- 2023M^2ConceptBase: A Fine-Grained Aligned Concept-Centric Multimodal Knowledge Basecited by 2
- 2025VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generationcited by 2
- 2025MMhops-R1: Multimodal Multi-hop Reasoningcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Theseus in the Graph: Towards Traceable Multi-Hop Graph Navigation
- 2025BYOKG-RAG: Multi-Strategy Graph Retrieval for Knowledge Graph Question Answering
- 2025StepChain GraphRAG: Reasoning Over Knowledge Graphs for Multi-Hop Question Answering
- 2026M3-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering
- 2025DAMR: Efficient and Adaptive Context-Aware Knowledge Graph Question Answering with LLM-Guided MCTS
- 2025Multimodal Multihop Source Retrieval for Web Question Answering
- 2026S-Path-RAG: Semantic-Aware Shortest-Path Retrieval Augmented Generation for Multi-Hop Knowledge Graph Question Answering
- 2025Search-on-Graph: Iterative Informed Navigation for Large Language Model Reasoning on Knowledge Graphs