# Noise-powered Multi-modal Knowledge Graph Representation Framework

arXiv: https://arxiv.org/abs/2403.06832 · HTML: https://mlsanity.com/paper/2403.06832/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 64 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Cross-lingual Entity Alignment via Joint Attribute-Preserving Embedding](https://mlsanity.com/paper/1708.05045/) (2017)
- [A Benchmarking Study of Embedding-based Entity Alignment for Knowledge Graphs](https://mlsanity.com/paper/2003.07743/) (2020)
- [MMKG: Multi-Modal Knowledge Graphs](https://mlsanity.com/paper/1903.05485/) (2019)
- [NEFTune: Noisy Embeddings Improve Instruction Finetuning](https://mlsanity.com/paper/2310.05914/) (2023)
- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph](https://mlsanity.com/paper/2309.13625/) (2023)
- [Image-embodied Knowledge Representation Learning](https://mlsanity.com/paper/1609.07028/) (2016)
- [Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey](https://mlsanity.com/paper/2402.05391/) (2024)
- [Visual Pivoting for (Unsupervised) Entity Alignment](https://mlsanity.com/paper/2009.13603/) (2020)
- [Aligning Cross-Lingual Entities with Multi-Aspect Information](https://mlsanity.com/paper/1910.06575/) (2019)
- [Multi-modal Contrastive Representation Learning for Entity Alignment](https://mlsanity.com/paper/2209.00891/) (2022)
- [Contrastive Language-Image Pre-Training with Knowledge Graphs](https://mlsanity.com/paper/2210.08901/) (2022)
- [Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment](https://mlsanity.com/paper/2307.16210/) (2023)
- [Multi-modal Entity Alignment in Hyperbolic Space](https://mlsanity.com/paper/2106.03619/) (2021)
- [Vision, Deduction and Alignment: An Empirical Study on Multi-modal Knowledge Graph Alignment](https://mlsanity.com/paper/2302.08774/) (2023)
- [Knowledge Graph Completion with Pre-trained Multimodal Transformer and Twins Negative Sampling](https://mlsanity.com/paper/2209.07084/) (2022)
- [Endowing Language Models with Multimodal Knowledge Graph Representations](https://mlsanity.com/paper/2206.13163/) (2022)
- [Revisit and Outstrip Entity Alignment: A Perspective of Generative Models](https://mlsanity.com/paper/2305.14651/) (2023)

## Built on by (most cited first, 7 of 7)

- [VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddings](https://mlsanity.com/paper/2603.02435/) (2026)
- [MyGram: Modality-aware Graph Transformer with Global Distribution for Multi-modal Entity Alignment](https://mlsanity.com/paper/2601.11885/) (2026)
- [Every Little Helps: Building Knowledge Graph Foundation Model with Fine-grained Transferable Multi-modal Tokens](https://mlsanity.com/paper/2602.15896/) (2026)
- [E-MMKGR: A Unified Multimodal Knowledge Graph Framework for E-commerce Applications](https://mlsanity.com/paper/2602.20877/) (2026)
- [Are a Thousand Words Better Than a Single Picture? Beyond Images -- A Framework for Multi-Modal Knowledge Graph Dataset Enrichment](https://mlsanity.com/paper/2603.16974/) (2026)
- [Time Imprint: Learning Time-Aware Representations in Multi-Modal Knowledge Graphs](https://mlsanity.com/paper/2607.09777/) (2026)
- [More Perspectives, Stronger Signals: Multi-Perspective Enhancement and Progressive Fusion for Multimodal Entity Representation Learning](https://mlsanity.com/paper/2608.29139/) (2026)

## Abstract (arXiv)

The rise of Multi-modal Pre-training highlights the necessity for a unified Multi-Modal Knowledge Graph (MMKG) representation learning framework. Such a framework is essential for embedding structured knowledge into multi-modal Large Language Models effectively, alleviating issues like knowledge misconceptions and multi-modal hallucinations. In this work, we explore the efficacy of models in accurately embedding entities within MMKGs through two pivotal tasks: Multi-modal Knowledge Graph Completion (MKGC) and Multi-modal Entity Alignment (MMEA). Building on this foundation, we propose a novel SNAG method that utilizes a Transformer-based architecture equipped with modality-level noise masking to robustly integrate multi-modal entity features in KGs. By incorporating specific training objectives for both MKGC and MMEA, our approach achieves SOTA performance across a total of ten datasets, demonstrating its versatility. Moreover, SNAG can not only function as a standalone model but also enhance other existing methods, providing stable performance improvements. Code and data are available at https://github.com/zjukg/SNAG.