mlsanityData to 7 Oct 2026

Multimodal Reasoning with Multimodal Knowledge Graph

arXiv 2406.02030 ↗First posted 4 Jun 2024cs.CL cs.AIcited by 43

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Multimodal reasoning with large language models (LLMs) often suffers from hallucinations and the presence of deficient or outdated knowledge within LLMs. Some approaches have sought to mitigate these issues by employing textual knowledge graphs, but their singular modality of knowledge limits comprehensive cross-modal understanding. In this paper, we propose the Multimodal Reasoning with Multimodal Knowledge Graph (MR-MKG) method, which leverages multimodal knowledge graphs (MMKGs) to learn rich and semantic knowledge across modalities, significantly enhancing the multimodal reasoning capabilities of LLMs. In particular, a relation graph attention network is utilized for encoding MMKGs and a cross-modal alignment module is designed for optimizing image-text alignment. A MMKG-grounded dataset is constructed to equip LLMs with initial expertise in multimodal reasoning through pretraining. Remarkably, MR-MKG achieves superior performance while training on only a small fraction of parameters, approximately 2.25% of the LLM's parameter size. Experimental results on multimodal question answering and multimodal analogy reasoning tasks demonstrate that our MR-MKG method outperforms previous state-of-the-art models.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 62 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

36 of 62 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning23 Jan
  2. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models8 Mar
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2017Graph Attention Networkscited by 8,854
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  12. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
21 more
  1. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  2. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  4. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  6. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  7. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  8. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  9. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  10. 2023Generating Images with Multimodal Language Modelscited by 259
  11. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  12. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  13. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  14. 2023Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answeringcited by 119
  15. 2019MMKG: Multi-Modal Knowledge Graphscited by 104
  16. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
  17. 2016Image-embodied Knowledge Representation Learningcited by 68
  18. 2023Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answeringcited by 64
  19. 2021Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graphcited by 31
  20. 2023Teaching Language Models to Hallucinate Less with Synthetic Taskscited by 31
  21. 2022E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoningcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 25; 2026: 15.

43 papers.

Sort:

By area:

  1. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  2. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  3. 2025MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teamingcited by 31
  4. 2024CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Modelscited by 29
  5. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 28
  6. 2025Mind with Eyes: from Language Reasoning to Multimodal Reasoningcited by 22
  7. 2025Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoningcited by 19
  8. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  9. 2025Safety Alignment via Constrained Knowledge Unlearningcited by 13
  10. 2025Weaving Context Across Images: Improving Vision-Language Models through Focus-Centric Visual Chainscited by 11
  11. 2026SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenescited by 11
  12. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 10
31 more
  1. 2025mKG-RAG: Leveraging Multimodal Knowledge Graphs in Retrieval-Augmented Generation for Knowledge-intensive VQAcited by 10
  2. 2025A Survey of Link Prediction in Temporal Networkscited by 6
  3. 2025MegaRAG: Multimodal Knowledge Graph-Based Retrieval Augmented Generationcited by 6
  4. 2025Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decouplingcited by 5
  5. 2025FineQuest: Adaptive Knowledge-Assisted Sports Video Understanding via Agent-of-Thoughts Reasoningcited by 5
  6. 2026MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledgecited by 5
  7. 2025MEDMKG: Benchmarking Medical Knowledge Exploitation with Multimodal Knowledge Graphcited by 2
  8. 2025VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generationcited by 2
  9. 2026VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddingscited by 2
  10. 2026When Modalities Remember: Continual Learning for Multimodal Knowledge Graphscited by 2
  11. 2025Diffusion-based Hierarchical Negative Sampling for Multimodal Knowledge Graph Completioncited by 1
  12. 2025COSINT-Agent: A Knowledge-Driven Multimodal Agent for Chinese Open Source Intelligencecited by 1
  13. 2025Evaluating Graphical Perception with Multimodal LLMscited by 1
  14. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Modelscited by 1
  15. 2026KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answeringcited by 1
  16. 2026MG2-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generationcited by 1
  17. 2026D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank Approximationcited by 1
  18. 2026A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understandingcited by 1
  19. 2025A Generative Framework for Bidirectional Image-Report Understanding in Chest Radiographycited by 0
  20. 2025Towards Multimodal Graph Large Language Modelcited by 0
  21. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0
  22. 2025FusionAdapter for Few-Shot Relation Learning in Multimodal Knowledge Graphscited by 0
  23. 2025MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Modelscited by 0
  24. 2025M3KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generationcited by 0
  25. 2026A Survey on Semantic Communication for Vision: Categories, Frameworks, Enabling Techniques, and Applicationscited by 0
  26. 2026Are a Thousand Words Better Than a Single Picture? Beyond Images -- A Framework for Multi-Modal Knowledge Graph Dataset Enrichmentcited by 0
  27. 2026Ontology-Guided Evidence Path Inference for Multi-hop Knowledge Graph Question Answeringcited by 0
  28. 2026Multimodal Graph RAG for Long-range Visually Rich Document Understandingcited by 0
  29. 2026Generative and multimodal AI for materials prediction and design: Progress, challenges, and perspectivescited by 0
  30. 2026HANIA: Planner-Guided Multimodal Graph Evidence Selection for Grounded Question Answeringcited by 0
  31. 2026Evolution of Multimodal Question Answering: From Modality-Adaptive Extraction to Unified Language Representationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mario: Multimodal Graph Reasoning with Large Language Models
  2. 2025Hybrid-DMKG: A Hybrid Reasoning Framework over Dynamic Multimodal Knowledge Graphs for Multimodal Multihop QA with Knowledge Editing
  3. 2022Multimodal Analogical Reasoning over Knowledge Graphs
  4. 2025Exploring and Evaluating Multimodal Knowledge Reasoning Consistency of Multimodal Large Language Models
  5. 2024Understanding the Role of LLMs in Multimodal Evaluation Benchmarks
  6. 2023Structure Guided Multi-modal Pre-trained Transformer for Knowledge Graph Reasoning
  7. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
  8. 2026MMCOMET: A Large-Scale Multimodal Commonsense Knowledge Graph for Contextual Reasoning