mlsanityData to 14 Oct 2026

Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey

arXiv 2402.05391 ↗First posted 8 Feb 2024cs.AI cs.CV cs.IRcited by 53

Not on the map. Why?

Abstract (arXiv)

Knowledge Graphs (KGs) play a pivotal role in advancing various AI applications, with the semantic web community's exploration into multi-modal dimensions unlocking new avenues for innovation. In this survey, we carefully review over 300 articles, focusing on KG-aware research in two principal aspects: KG-driven Multi-Modal (KG4MM) learning, where KGs support multi-modal tasks, and Multi-Modal Knowledge Graph (MM4KG), which extends KG studies into the MMKG realm. We begin by defining KGs and MMKGs, then explore their construction progress. Our review includes two primary task categories: KG-aware multi-modal learning tasks, such as Image Classification and Visual Question Answering, and intrinsic MMKG tasks like Multi-modal Knowledge Graph Completion and Entity Alignment, highlighting specific research trajectories. For most of these tasks, we provide definitions, evaluation benchmarks, and additionally outline essential insights for conducting relevant research. Finally, we discuss current challenges and identify emerging trends, such as progress in Large Language Modeling and Multi-modal Pre-training strategies. This survey aims to serve as a comprehensive reference for researchers already involved in or considering delving into KG and multi-modal learning research, offering insights into the evolving landscape of MMKG research and supporting future work.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 166 of its 611 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

166 of 611 references.

References · 166

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  10. 2017Graph Attention Networkscited by 8,923
  11. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  12. 2023Segment Anythingcited by 8,234
38 more
  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2016Improved Techniques for Training GANscited by 2,970
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2018YOLOv3: An Incremental Improvementcited by 2,659
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  13. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  14. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  15. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  16. 2019RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Spacecited by 898
  17. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  18. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  19. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  20. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  21. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  22. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  23. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  24. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  25. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  26. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  27. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  28. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  29. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  30. 2019KG-BERT: BERT for Knowledge Graph Completioncited by 324
  31. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 302
  32. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  33. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 298
  34. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 238
  35. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  36. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 178
  37. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  38. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 168

50 most cited of 166 All 166 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 23; 2025: 16; 2026: 14.

53 papers.

Sort:

By area:

  1. 2024SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generationcited by 34
  2. 2024Mosaic of Modalities: A Comprehensive Benchmark for Multimodal Graph Learningcited by 29
  3. 2024NativE: Multi-modal Knowledge Graph Completion in the Wildcited by 23
  4. 2024An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Modelscited by 22
  5. 2025Mind with Eyes: from Language Reasoning to Multimodal Reasoningcited by 22
  6. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveyscited by 20
  7. 2025Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoningcited by 19
  8. 2025Graph4MM: Weaving Multimodal Learning with Structural Informationcited by 17
  9. 2024Modality-Aware Integration with Large Language Models for Knowledge-based Visual Question Answeringcited by 16
  10. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  11. 2024Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learningcited by 13
  12. 2024Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completioncited by 10
38 more
  1. 2024Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphscited by 10
  2. 2024Towards Artwork Explanation in Large-scale Vision Language Modelscited by 8
  3. 2024Multi-domain Knowledge Graph Collaborative Pre-training and Prompt Tuning for Diverse Downstream Taskscited by 8
  4. 2024Noise-powered Multi-modal Knowledge Graph Representation Frameworkcited by 7
  5. 2024Towards Cross-Lingual Explanation of Artwork in Large-scale Vision Language Modelscited by 7
  6. 2024Efficient and Effective Adaptation of Multimodal Foundation Models in Sequential Recommendationcited by 7
  7. 2024ASGEA: Exploiting Logic Rules from Align-Subgraphs for Entity Alignmentcited by 5
  8. 2024UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Modelscited by 5
  9. 2026ContextRL: Enhancing MLLM's Knowledge Discovery Efficiency with Context-Augmented RLcited by 5
  10. 2025Taming the Untamed: Graph-Based Knowledge Retrieval and Reasoning for MLLMs to Conquer the Unknowncited by 4
  11. 2024DRAK: Unlocking Molecular Insights with Domain-Specific Retrieval-Augmented Knowledge in LLMscited by 3
  12. 2025Abstractive Visual Understanding of Multi-modal Structured Knowledge: A New Perspective for MLLM Evaluationcited by 3
  13. 2024Atomas: Hierarchical Alignment on Molecule-Text for Unified Molecule Understanding and Generationcited by 2
  14. 2024Exploring Graph Mamba: A Comprehensive Survey on State-Space Models for Graph Learningcited by 2
  15. 2025LLM-Powered Knowledge Graphs for Enterprise Intelligence and Analyticscited by 2
  16. 2025Leveraging Knowledge Graphs and LLMs for Context-Aware Messagingcited by 2
  17. 2025Med-CRAFT: An Information System for Explainable and Configurable Construction of Multimodal Medical QA Datasetscited by 2
  18. 2026When Modalities Remember: Continual Learning for Multimodal Knowledge Graphscited by 2
  19. 2024IRR: Image Review Ranking Framework for Evaluating Vision-Language Modelscited by 1
  20. 2024The Master-Slave Encoder Model for Improving Patent Text Summarization: A New Approach to Combining Specifications and Claimscited by 1
  21. 2025StaR-KVQA: Structured Reasoning Traces for Implicit-Knowledge Visual Question Answeringcited by 1
  22. 2026Multi-Perspective Evidence Synthesis and Reasoning for Unsupervised Multimodal Entity Linkingcited by 1
  23. 2026Knowledge Graph-Driven Expert-Level Reasoning for Neurosciencecited by 1
  24. 2026Multi-Granularity Context-Enhanced RAG over Multimodal Knowledge Graphscited by 1
  25. 2024A Pattern to Align Them All: Integrating Different Modalities to Define Multi-Modal Entitiescited by 0
  26. 2025Multi-modal Knowledge Graph Generation with Semantics-enriched Promptscited by 0
  27. 2025A Survey of Task-Oriented Knowledge Graph Reasoning: Status, Applications, and Prospectscited by 0
  28. 2025I2CR: Intra- and Inter-modal Collaborative Reflections for Multimodal Entity Linkingcited by 0
  29. 2025NLKI: A lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Taskscited by 0
  30. 2025Collaboration of Fusion and Independence: Hypercomplex-driven Robust Multi-Modal Knowledge Graph Completioncited by 0
  31. 2025FusionAdapter for Few-Shot Relation Learning in Multimodal Knowledge Graphscited by 0
  32. 2025VisKnow: Constructing Visual Knowledge Base for Object Understandingcited by 0
  33. 2026Every Little Helps: Building Knowledge Graph Foundation Model with Fine-grained Transferable Multi-modal Tokenscited by 0
  34. 2026CAMD: Coverage-Aware Multimodal Decoding for Efficient Reasoning of Multimodal Large Language Modelscited by 0
  35. 2026Are a Thousand Words Better Than a Single Picture? Beyond Images -- A Framework for Multi-Modal Knowledge Graph Dataset Enrichmentcited by 0
  36. 2026RADD: Retrieval-Augmented Discrete Diffusion for Multi-Modal Knowledge Graph Completioncited by 0
  37. 2026Evidence-Based Intelligent Diagnostic and Therapeutic Visualization System with Large Language Models: Multi-Turn Interaction and Multimodal Treatment Plan Generationcited by 0
  38. 2026More Perspectives, Stronger Signals: Multi-Perspective Enhancement and Progressive Fusion for Multimodal Entity Representation Learningcited by 0

50 most cited of 53 All 53 (JSON)