mlsanityData to 7 Oct 2026

Are a Thousand Words Better Than a Single Picture? Beyond Images -- A Framework for Multi-Modal Knowledge Graph Dataset Enrichment

arXiv 2603.16974 ↗First posted 17 Mar 2026cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Multi-Modal Knowledge Graphs (MMKGs) benefit from visual information, yet large-scale image collection is hard to curate and often excludes ambiguous but relevant visuals (e.g., logos, symbols, abstract scenes). We present Beyond Images, an automatic data-centric enrichment pipeline with optional human auditing. This pipeline operates in three stages: (1) large-scale retrieval of additional entity-related images, (2) conversion of all visual inputs into textual descriptions to ensure that ambiguous images contribute usable semantics rather than noise, and (3) fusion of multi-source descriptions using a large language model (LLM) to generate concise, entity-aligned summaries. These summaries replace or augment the text modality in standard MMKG models without changing their architectures or loss functions. Across three public MMKG datasets and multiple baseline models, we observe consistent gains (up to 7% Hits@1 overall). Furthermore, on a challenging subset of entities with visually ambiguous logos and symbols, converting images into text yields large improvements (201.35% MRR and 333.33% Hits@1). Additionally, we release a lightweight Text-Image Consistency Check Interface for optional targeted audits, improving description quality and dataset reliability. Our results show that scaling image coverage and converting ambiguous visuals into text is a practical path to stronger MMKG completion. Code, datasets, and supplementary materials are available at https://github.com/pengyu-zhang/Beyond-Images.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 35 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

28 of 35 references.

References · 28

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  3. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  4. 2023Temporal Graph Benchmark for Machine Learning on Temporal Graphscited by 137
  5. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  6. 2019MMKG: Multi-Modal Knowledge Graphscited by 104
  7. 2015Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labelscited by 64
  8. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  9. 2017BAM! The Behance Artistic Media Dataset for Recognition Beyond Photographycited by 43
  10. 2024Multimodal Reasoning with Multimodal Knowledge Graphcited by 43
  11. 2022Multi-modal Contrastive Representation Learning for Entity Alignmentcited by 37
  12. 2021Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graphcited by 31
16 more
  1. 2024SAC-KG: Exploiting Large Language Models as Skilled Automatic Constructors for Domain Knowledge Graphscited by 31
  2. 2023IMF: Interactive Multimodal Fusion Model for Link Predictioncited by 28
  3. 2022Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioningcited by 25
  4. 2024NativE: Multi-modal Knowledge Graph Completion in the Wildcited by 23
  5. 2019Logo-2K+: A Large-Scale Logo Dataset for Scalable Logo Classificationcited by 22
  6. 2021OntoEA: Ontology-guided Entity Alignment via Joint Knowledge Graph Embeddingcited by 19
  7. 2025The Power of Context: How Multimodality Improves Image Super-Resolutioncited by 15
  8. 2024Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representationcited by 14
  9. 2021Edge: Enriching Knowledge Graph Embeddings with External Textcited by 13
  10. 2025FakeScope: Large Multimodal Expert Model for Transparent AI-Generated Image Forensicscited by 11
  11. 2024Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completioncited by 10
  12. 2024Noise-powered Multi-modal Knowledge Graph Representation Frameworkcited by 7
  13. 2024AutoML-guided Fusion of Entity and LLM-based Representations for Document Classificationcited by 3
  14. 2024LoginMEA: Local-to-Global Interaction Network for Multi-modal Entity Alignmentcited by 2
  15. 2025ReaLitE: Enrichment of Relation Embeddings in Knowledge Graphs using Numeric Literalscited by 2
  16. 2025Ambiguity-Aware and High-Order Relation Learning for Multi-Grained Image-Text Matchingcited by 1

Built on by

Not cited yet.