# Multi-modal Knowledge Graph Generation with Semantics-enriched Prompts

arXiv: https://arxiv.org/abs/2504.13631 · HTML: https://mlsanity.com/paper/2504.13631/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 18 Apr 2025.

## Builds on (most cited first, 14 of 14)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Composition-based Multi-Relational Graph Convolutional Networks](https://mlsanity.com/paper/1911.03082/) (2019)
- [Unsupervised Semantic Correspondence Using Stable Diffusion](https://mlsanity.com/paper/2305.15581/) (2023)
- [MMKG: Multi-Modal Knowledge Graphs](https://mlsanity.com/paper/1903.05485/) (2019)
- [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023)
- [Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey](https://mlsanity.com/paper/2402.05391/) (2024)
- [Prompt Engineering for Healthcare: Methodologies and Applications](https://mlsanity.com/paper/2304.14670/) (2023)
- [VisualSem: A High-quality Knowledge Graph for Vision and Language](https://mlsanity.com/paper/2008.09150/) (2020)
- [NativE: Multi-modal Knowledge Graph Completion in the Wild](https://mlsanity.com/paper/2406.17605/) (2024)
- [AspectMMKG: A Multi-modal Knowledge Graph with Aspect-aware Entities](https://mlsanity.com/paper/2308.04992/) (2023)

## Abstract (arXiv)

Multi-modal Knowledge Graphs (MMKGs) have been widely applied across various domains for knowledge representation. However, the existing MMKGs are significantly fewer than required, and their construction faces numerous challenges, particularly in ensuring the selection of high-quality, contextually relevant images for knowledge graph enrichment. To address these challenges, we present a framework for constructing MMKGs from conventional KGs. Furthermore, to generate higher-quality images that are more relevant to the context in the given knowledge graph, we designed a neighbor selection method called Visualizable Structural Neighbor Selection (VSNS). This method consists of two modules: Visualizable Neighbor Selection (VNS) and Structural Neighbor Selection (SNS). The VNS module filters relations that are difficult to visualize, while the SNS module selects neighbors that most effectively capture the structural characteristics of the entity. To evaluate the quality of the generated images, we performed qualitative and quantitative evaluations on two datasets, MKG-Y and DB15K. The experimental results indicate that using the VSNS method to select neighbors results in higher-quality images that are more relevant to the knowledge graph.