# Are a Thousand Words Better Than a Single Picture? Beyond Images -- A Framework for Multi-Modal Knowledge Graph Dataset Enrichment

arXiv: https://arxiv.org/abs/2603.16974 · HTML: https://mlsanity.com/paper/2603.16974/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 35 references that are in the map.
- First posted on arXiv 17 Mar 2026.

## Builds on (most cited first, 28 of 28)

- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [MoE-LLaVA: Mixture of Experts for Large Vision-Language Models](https://mlsanity.com/paper/2401.15947/) (2024)
- [Temporal Graph Benchmark for Machine Learning on Temporal Graphs](https://mlsanity.com/paper/2307.01026/) (2023)
- [From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models](https://mlsanity.com/paper/2212.10846/) (2022)
- [MMKG: Multi-Modal Knowledge Graphs](https://mlsanity.com/paper/1903.05485/) (2019)
- [Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels](https://mlsanity.com/paper/1512.06974/) (2015)
- [Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey](https://mlsanity.com/paper/2402.05391/) (2024)
- [Multimodal Reasoning with Multimodal Knowledge Graph](https://mlsanity.com/paper/2406.02030/) (2024)
- [BAM! The Behance Artistic Media Dataset for Recognition Beyond Photography](https://mlsanity.com/paper/1704.08614/) (2017)
- [Multi-modal Contrastive Representation Learning for Entity Alignment](https://mlsanity.com/paper/2209.00891/) (2022)
- [Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph](https://mlsanity.com/paper/2107.11970/) (2021)
- [SAC-KG: Exploiting Large Language Models as Skilled Automatic Constructors for Domain Knowledge Graphs](https://mlsanity.com/paper/2410.02811/) (2024)
- [IMF: Interactive Multimodal Fusion Model for Link Prediction](https://mlsanity.com/paper/2303.10816/) (2023)
- [Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning](https://mlsanity.com/paper/2205.04363/) (2022)
- [NativE: Multi-modal Knowledge Graph Completion in the Wild](https://mlsanity.com/paper/2406.17605/) (2024)
- [Logo-2K+: A Large-Scale Logo Dataset for Scalable Logo Classification](https://mlsanity.com/paper/1911.07924/) (2019)
- [OntoEA: Ontology-guided Entity Alignment via Joint Knowledge Graph Embedding](https://mlsanity.com/paper/2105.07688/) (2021)
- [The Power of Context: How Multimodality Improves Image Super-Resolution](https://mlsanity.com/paper/2503.14503/) (2025)
- [Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation](https://mlsanity.com/paper/2404.09468/) (2024)
- [Edge: Enriching Knowledge Graph Embeddings with External Text](https://mlsanity.com/paper/2104.04909/) (2021)
- [FakeScope: Large Multimodal Expert Model for Transparent AI-Generated Image Forensics](https://mlsanity.com/paper/2503.24267/) (2025)
- [Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion](https://mlsanity.com/paper/2402.15444/) (2024)
- [Noise-powered Multi-modal Knowledge Graph Representation Framework](https://mlsanity.com/paper/2403.06832/) (2024)
- [AutoML-guided Fusion of Entity and LLM-based Representations for Document Classification](https://mlsanity.com/paper/2408.09794/) (2024)
- [LoginMEA: Local-to-Global Interaction Network for Multi-modal Entity Alignment](https://mlsanity.com/paper/2407.19625/) (2024)
- [ReaLitE: Enrichment of Relation Embeddings in Knowledge Graphs using Numeric Literals](https://mlsanity.com/paper/2504.00852/) (2025)
- [Ambiguity-Aware and High-Order Relation Learning for Multi-Grained Image-Text Matching](https://mlsanity.com/paper/2507.09256/) (2025)

## Abstract (arXiv)

Multi-Modal Knowledge Graphs (MMKGs) benefit from visual information, yet large-scale image collection is hard to curate and often excludes ambiguous but relevant visuals (e.g., logos, symbols, abstract scenes). We present Beyond Images, an automatic data-centric enrichment pipeline with optional human auditing. This pipeline operates in three stages: (1) large-scale retrieval of additional entity-related images, (2) conversion of all visual inputs into textual descriptions to ensure that ambiguous images contribute usable semantics rather than noise, and (3) fusion of multi-source descriptions using a large language model (LLM) to generate concise, entity-aligned summaries. These summaries replace or augment the text modality in standard MMKG models without changing their architectures or loss functions. Across three public MMKG datasets and multiple baseline models, we observe consistent gains (up to 7% Hits@1 overall). Furthermore, on a challenging subset of entities with visually ambiguous logos and symbols, converting images into text yields large improvements (201.35% MRR and 333.33% Hits@1). Additionally, we release a lightweight Text-Image Consistency Check Interface for optional targeted audits, improving description quality and dataset reliability. Our results show that scaling image coverage and converting ambiguous visuals into text is a practical path to stronger MMKG completion. Code, datasets, and supplementary materials are available at https://github.com/pengyu-zhang/Beyond-Images.