mlsanityData to 7 Oct 2026

KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering

arXiv 2601.11632 ↗First posted 14 Jan 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Multi-modal Large Language Models (MLLMs) for Visual Question Answering (VQA) often suffer from dual limitations: knowledge hallucination and insufficient fine-grained visual perception. Crucially, we identify that commonsense graphs and scene graphs provide precisely complementary solutions to these respective deficiencies by providing rich external knowledge and capturing fine-grained visual details. However, prior works typically treat them in isolation, overlooking their synergistic potential. To bridge this gap, we propose KG-ViP, a unified framework that empowers MLLMs by fusing scene graphs and commonsense graphs. The core of the KG-ViP framework is a novel retrieval-and-fusion pipeline that utilizes the query as a semantic bridge to progressively integrate both graphs, synthesizing a unified structured context that facilitates reliable multi-modal reasoning. Extensive experiments on FVQA 2.0+ and MVQA benchmarks demonstrate that KG-ViP significantly outperforms existing VQA methods.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 59 references that are in the map. First posted on arXiv 14 Jan 2026.

Builds on

27 of 59 references.

References · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  3. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  4. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  5. 2024A Survey on LLM-as-a-Judgecited by 1,384
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  8. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  9. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 235
  10. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  11. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  12. 2021A Comprehensive Survey of Scene Graphs: Generation and Applicationcited by 125
15 more
  1. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  2. 2020A Benchmarking Study of Embedding-based Entity Alignment for Knowledge Graphscited by 119
  3. 2019MMKG: Multi-Modal Knowledge Graphscited by 104
  4. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 93
  5. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  6. 2022Scene Graph Generation: A Comprehensive Surveycited by 73
  7. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 61
  8. 2021Semantic Answer Similarity for Evaluating Question Answering Modelscited by 47
  9. 2024Multimodal Reasoning with Multimodal Knowledge Graphcited by 43
  10. 2024DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Expertscited by 42
  11. 2024MovieBench: A Hierarchical Movie Level Dataset for Long Video Generationcited by 35
  12. 2024Efficient Algorithms for Personalized PageRank Computation: A Surveycited by 21
  13. 2024Deliberate Reasoning in Language Models as Structure-Aware Planning with an Accurate World Modelcited by 20
  14. 2023Combo of Thinking and Observing for Outside-Knowledge VQAcited by 9
  15. 2023GPT4SGG: Synthesizing Scene Graphs from Holistic and Region-specific Narrativescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Adaptive Information Control for Search-Augmented LLM Reasoningcited by 8

Similar, not cited

Close in wording, no citation either way

  1. 2024Find The Gap: Knowledge Base Reasoning For Visual Question Answering
  2. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  3. 2025MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering
  4. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
  5. 2023A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answering
  6. 2025See the Forest and the Trees: A Synergistic Reasoning Framework for Knowledge-Based Visual Question Answering
  7. 2026MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answering
  8. 2024Precision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Models