mlsanityData to 7 Oct 2026

Multi-Modal Knowledge Graph Construction and Application: A Survey

arXiv 2202.05786 ↗First posted 11 Feb 2022cs.AI cs.CL cs.CVcited by 74

Not on the map. Why?

Abstract (arXiv)

Recent years have witnessed the resurgence of knowledge engineering which is featured by the fast growth of knowledge graphs. However, most of existing knowledge graphs are represented with pure symbols, which hurts the machine's capability to understand the real world. The multi-modalization of knowledge graphs is an inevitable key step towards the realization of human-level machine intelligence. The results of this endeavor are Multi-modal Knowledge Graphs (MMKGs). In this survey on MMKGs constructed by texts and images, we first give definitions of MMKGs, followed with the preliminaries on multi-modal tasks and techniques. We then systematically review the challenges, progresses and opportunities on the construction and application of MMKGs respectively, with detailed analyses of the strength and weakness of different solutions. We finalize this survey with open research problems relevant to MMKGs.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Builds on 91 of its 207 references that are in the map. First posted on arXiv 11 Feb 2022.

Builds on

91 of 207 references.

References · 91

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
38 more
  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  2. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  3. 2021Grounded Language-Image Pre-trainingcited by 1,147
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  5. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  6. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  7. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  8. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  9. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  10. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  11. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  12. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  13. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  14. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  15. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  16. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
  17. 2016Visual Relationship Detection with Language Priorscited by 333
  18. 2017The (Un)reliability of saliency methodscited by 331
  19. 2016Top-down Neural Attention by Excitation Backpropcited by 318
  20. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  21. 2018Graph R-CNN for Scene Graph Generationcited by 296
  22. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  23. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 258
  24. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 254
  25. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  26. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  27. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  28. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 197
  29. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  30. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  31. 2016Visual Storytellingcited by 176
  32. 2019Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchycited by 156
  33. 2017Dual-Path Convolutional Image-Text Embeddings with Instance Losscited by 145
  34. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  35. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  36. 2017Visual Translation Embedding Network for Visual Relation Detectioncited by 112
  37. 2019CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrievalcited by 110
  38. 2019MMKG: Multi-Modal Knowledge Graphscited by 104

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 18; 2024: 19; 2025: 19; 2026: 10.

74 papers.

Sort:

By area:

  1. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  4. 2022A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodalcited by 106
  5. 2023LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunitiescited by 97
  6. 2023Knowledge Graphs: Opportunities and Challengescited by 70
  7. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  8. 2022Generative Knowledge Graph Construction: A Reviewcited by 49
  9. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  10. 2023Attribute-Consistent Knowledge Graph Representation Learning for Multi-Modal Entity Alignmentcited by 23
  11. 2023KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graphcited by 22
  12. 2025Mind with Eyes: from Language Reasoning to Multimodal Reasoningcited by 22
38 more
  1. 2022Multimodal Analogical Reasoning over Knowledge Graphscited by 21
  2. 2024Multi-perspective Improvement of Knowledge Graph Completion with Large Language Modelscited by 20
  3. 2025Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoningcited by 19
  4. 2023Scene-Driven Multimodal Knowledge Graph Construction for Embodied AIcited by 17
  5. 2024UMIE: Unified Multimodal Information Extraction with Instruction Tuningcited by 17
  6. 2025Graph4MM: Weaving Multimodal Learning with Structural Informationcited by 17
  7. 2025MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphscited by 16
  8. 2025Deep Learning and Foundation Models for Weather Prediction: A Surveycited by 15
  9. 2022MoSE: Modality Split and Ensemble for Multimodal Knowledge Graph Completioncited by 14
  10. 2023UKnow: A Unified Knowledge Protocol with Multimodal Knowledge Graph Datasets for Reasoning and Vision-Language Pre-Trainingcited by 14
  11. 2023Construction of Knowledge Graphs: State and Challengescited by 12
  12. 2025From Screens to Scenes: A Survey of Embodied AI in Healthcarecited by 10
  13. 2022Kuaipedia: a Large-scale Multi-modal Short-video Encyclopediacited by 8
  14. 2023Dual-Gated Fusion with Prefix-Tuning for Multi-Modal Relation Extractioncited by 7
  15. 2024Noise-powered Multi-modal Knowledge Graph Representation Frameworkcited by 7
  16. 2023Multi-Modal Knowledge Graph Transformer Framework for Multi-Modal Entity Alignmentcited by 6
  17. 2025Biomedical Knowledge Graph: A Survey of Domains, Tasks, and Real-World Applicationscited by 6
  18. 2023AspectMMKG: A Multi-modal Knowledge Graph with Aspect-aware Entitiescited by 5
  19. 2024ASGEA: Exploiting Logic Rules from Align-Subgraphs for Entity Alignmentcited by 5
  20. 2025A Survey of Link Prediction in N-ary Knowledge Graphscited by 4
  21. 2023Survey on Foundation Models for Prognostics and Health Management in Industrial Cyber-Physical Systemscited by 3
  22. 2022Dual Embodied-Symbolic Concept Representations for Deep Learningcited by 2
  23. 2023DiscoverPath: A Knowledge Refinement and Retrieval System for Interdisciplinarity on Biomedical Researchcited by 2
  24. 2023Retail-786k: a Large-Scale Dataset for Visual Entity Matchingcited by 2
  25. 2024Domain Adaptation for Large-Vocabulary Object Detectorscited by 2
  26. 2024Knowledge Guided Entity-aware Video Captioning and A Basketball Benchmarkcited by 2
  27. 2024Metarobotics for Industry and Society: Vision, Technologies, and Opportunitiescited by 2
  28. 2024RConE: Rough Cone Embedding for Multi-Hop Logical Query Answering on Multi-Modal Knowledge Graphscited by 2
  29. 2024Network Topology Inference from Smooth Signals Under Partial Observabilitycited by 2
  30. 2025Perovskite-LLM: Knowledge-Enhanced Large Language Models for Perovskite Solar Cell Researchcited by 2
  31. 2025DSRAG: A Domain-Specific Retrieval Framework Based on Document-derived Multimodal Knowledge Graphcited by 2
  32. 2025Learning with Dual-level Noisy Correspondence for Multi-modal Entity Alignmentcited by 2
  33. 2025Med-CRAFT: An Information System for Explainable and Configurable Construction of Multimodal Medical QA Datasetscited by 2
  34. 2026When Modalities Remember: Continual Learning for Multimodal Knowledge Graphscited by 2
  35. 2023Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion?cited by 1
  36. 2024Variational Multi-Modal Hypergraph Attention Network for Multi-Modal Relation Extractioncited by 1
  37. 2024Contrast then Memorize: Semantic Neighbor Retrieval-Enhanced Inductive Multimodal Knowledge Graph Completioncited by 1
  38. 2024wav2graph: A Framework for Supervised Learning Knowledge Graph from Speechcited by 1

50 most cited of 74 All 74 (JSON)