mlsanityData to 14 Oct 2026

MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs

arXiv 2507.20804 ↗First posted 28 Jul 2025cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) suffer from hallucinations due to their static parametric knowledge. Retrieval-Augmented Generation (RAG) and GraphRAG mitigate this issue by incorporating external knowledge and structured reasoning over knowledge graphs (KGs). However, existing approaches remain largely text-centric, as constructing fine-grained multimodal knowledge graphs (MMKGs) with explicit cross-modal semantics remains challenging. In this paper, we propose MMGraphRAG, a framework for building interpretable MMKGs that unify textual and visual knowledge. Our approach represents visual content as structured scene graphs and integrates them with textual KGs through a novel cross-modal entity linking method, SpecLink, which leverages spectral clustering to jointly model semantic similarity and graph structure. This design preserves explicit entities, relations, and reasoning paths across modalities, enabling structure-aware retrieval and generation. To support evaluation, we introduce the CMEL dataset, a benchmark for fine-grained cross-modal entity alignment. Experimental results on CMEL demonstrate improved entity linking accuracy, while evaluations on DocBench and MMLongBench show that MMGraphRAG achieves superior performance and stronger robustness, particularly in complex multimodal reasoning scenarios.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 41 of its 86 references that are in the map. First posted on arXiv 28 Jul 2025.

Builds on

41 of 86 references.

References · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  8. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  10. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  11. 2020Recipes for building an open-domain chatbotcited by 593
  12. 2017Scene Graph Generation by Iterative Message Passingcited by 462
29 more
  1. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 414
  2. 2018From Louvain to Leiden: guaranteeing well-connected communitiescited by 385
  3. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 356
  4. 2024Graph Retrieval-Augmented Generation: A Surveycited by 324
  5. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 267
  6. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  7. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  8. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 195
  9. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 183
  10. 2020GPS-Net: Graph Property Sensing Network for Scene Graph Generationcited by 166
  11. 2024Jasper and Stella: distillation of SOTA embedding modelscited by 112
  12. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 111
  13. 2019MMKG: Multi-Modal Knowledge Graphscited by 105
  14. 2021Energy-Based Learning for Scene Graph Generationcited by 104
  15. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 85
  16. 2024Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generationcited by 80
  17. 2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
  18. 2024StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurizationcited by 71
  19. 2024DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systemscited by 37
  20. 2025HM-RAG: Hierarchical Multi-Agent Multimodal Retrieval Augmented Generationcited by 31
  21. 2021Entity Linking Meets Deep Learning: Techniques and Solutionscited by 22
  22. 2025MMKB-RAG: A Multi-Modal Knowledge-Based Retrieval-Augmented Generation Frameworkcited by 18
  23. 2025Vision-Language Models Struggle to Align Entities across Modalitiescited by 14
  24. 2023A Dual-way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linkingcited by 12
  25. 2020Improving Candidate Generation for Low-resource Cross-lingual Entity Linkingcited by 10
  26. 2024OneNet: A Fine-Tuning Free Framework for Few-Shot Entity Linking via Large Language Model Promptingcited by 9
  27. 2023AMELI: Enhancing Multimodal Entity Linking with Fine-Grained Attributescited by 8
  28. 2025Reducing Hallucinations of Medical Multimodal Large Language Models with Visual Retrieval-Augmented Generationcited by 8
  29. 2024LLaVA-SG: Leveraging Scene Graphs as Visual Semantic Expression in Vision-Language Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 13.

16 papers.

Sort:

By area:

  1. 2025RAG-Anything: All-in-One RAG Frameworkcited by 20
  2. 2025mKG-RAG: Leveraging Multimodal Knowledge Graphs in Retrieval-Augmented Generation for Knowledge-intensive VQAcited by 11
  3. 2026Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baselinecited by 8
  4. 2025Structured and Abstractive Reasoning on Multi-modal Relational Knowledge Imagescited by 2
  5. 2026EvoGraph-R1: Self-Evolving Multimodal Knowledge Hypergraphs for Agentic Retrievalcited by 2
  6. 2026G2-Reader: Dual Evolving Graphs for Multimodal Document QAcited by 1
  7. 2026MG2-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generationcited by 1
  8. 2026FLOWREADER: Min-Cost Flow Optimization for Multi-Modal Long Document Q&Acited by 1
  9. 2026HVM-GraphRAG: Holistic-View Multimodal Graph Retrieval-Augmented Generation on Complex Documentcited by 1
  10. 2026Multi-Granularity Context-Enhanced RAG over Multimodal Knowledge Graphscited by 1
  11. 2026Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognitioncited by 0
  12. 2026AutothinkRAG: Complexity-Aware Control of Retrieval-Augmented Reasoning for Image-Text Interactioncited by 0
4 more
  1. 2026Task-Adaptive Retrieval over Agentic Multi-Modal Web Histories via Learned Graph Memorycited by 0
  2. 2026TAP-RAG: Task-Aware Policy Control for Long-Document Multimodal Question Answeringcited by 0
  3. 2026DeCoRAG: Cognitive Decoupling and Semantic-Aware Cropping for Complex Document Understandingcited by 0
  4. 2026DualG-MRAG: Decoupling Macro-Reasoning and Micro-Matching for Multimodal Retrieval-Augmented Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning
  2. 2026VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddings
  3. 2025VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generation
  4. 2025ELMM: Efficient Lightweight Multimodal Large Language Models for Multimodal Knowledge Graph Completion
  5. 2025DeepMEL: A Multi-Agent Collaboration Framework for Multimodal Entity Linking
  6. 2025Transformer-Based Multimodal Knowledge Graph Completion with Link-Aware Contexts
  7. 2025KGMEL: Knowledge Graph-Enhanced Multimodal Entity Linking
  8. 2024UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Models