# Multi-Modal Knowledge Graph Transformer Framework for Multi-Modal Entity Alignment

arXiv: https://arxiv.org/abs/2310.06365 · HTML: https://mlsanity.com/paper/2310.06365/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 10 Oct 2023.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [KGAT: Knowledge Graph Attention Network for Recommendation](https://mlsanity.com/paper/1905.07854/) (2019)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [MMKG: Multi-Modal Knowledge Graphs](https://mlsanity.com/paper/1903.05485/) (2019)
- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [Where To Look: Focus Regions for Visual Question Answering](https://mlsanity.com/paper/1511.07394/) (2015)
- [Exploring and Evaluating Attributes, Values, and Structures for Entity Alignment](https://mlsanity.com/paper/2010.03249/) (2020)
- [Visual Pivoting for (Unsupervised) Entity Alignment](https://mlsanity.com/paper/2009.13603/) (2020)
- [Multi-Modal Cross-Domain Alignment Network for Video Moment Retrieval](https://mlsanity.com/paper/2209.11572/) (2022)
- [You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videos](https://mlsanity.com/paper/2303.07863/) (2023)
- [Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries](https://mlsanity.com/paper/2208.07638/) (2022)
- [A Survey on Cross-Lingual Summarization](https://mlsanity.com/paper/2203.12515/) (2022)
- [Hierarchical Local-Global Transformer for Temporal Sentence Grounding](https://mlsanity.com/paper/2208.14882/) (2022)
- [Multi-modal Contrastive Representation Learning for Entity Alignment](https://mlsanity.com/paper/2209.00891/) (2022)
- [Kformer: Knowledge Injection in Transformer Feed-Forward Layers](https://mlsanity.com/paper/2201.05742/) (2022)
- [MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid](https://mlsanity.com/paper/2212.14454/) (2022)
- [Attribute-Consistent Knowledge Graph Representation Learning for Multi-Modal Entity Alignment](https://mlsanity.com/paper/2304.01563/) (2023)
- [Multi-modal Entity Alignment in Hyperbolic Space](https://mlsanity.com/paper/2106.03619/) (2021)
- [Vision, Deduction and Alignment: An Empirical Study on Multi-modal Knowledge Graph Alignment](https://mlsanity.com/paper/2302.08774/) (2023)
- [Towards Unifying Multi-Lingual and Cross-Lingual Summarization](https://mlsanity.com/paper/2305.09220/) (2023)
- [Transformer-based Entity Typing in Knowledge Graphs](https://mlsanity.com/paper/2210.11151/) (2022)
- [AliMe MKG: A Multi-modal Knowledge Graph for Live-streaming E-commerce](https://mlsanity.com/paper/2109.07411/) (2021)
- [Cross-Domain Aspect Extraction using Transformers Augmented with Knowledge Graphs](https://mlsanity.com/paper/2210.10144/) (2022)

## Built on by (most cited first, 7 of 7)

- [Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey](https://mlsanity.com/paper/2402.05391/) (2024)
- [Noise-powered Multi-modal Knowledge Graph Representation Framework](https://mlsanity.com/paper/2403.06832/) (2024)
- [LoginMEA: Local-to-Global Interaction Network for Multi-modal Entity Alignment](https://mlsanity.com/paper/2407.19625/) (2024)
- [MyGram: Modality-aware Graph Transformer with Global Distribution for Multi-modal Entity Alignment](https://mlsanity.com/paper/2601.11885/) (2026)
- [E-MMKGR: A Unified Multimodal Knowledge Graph Framework for E-commerce Applications](https://mlsanity.com/paper/2602.20877/) (2026)
- [Knowing When to Trust Images: Reliability-Aware Multi-modal Entity Alignment](https://mlsanity.com/paper/2609.23267/) (2026)
- [Silence and Noise: Resolving Suppression and Overvaluation in Multimodal Entity Alignment](https://mlsanity.com/paper/2610.16106/) (2026)

## Abstract (arXiv)

Multi-Modal Entity Alignment (MMEA) is a critical task that aims to identify equivalent entity pairs across multi-modal knowledge graphs (MMKGs). However, this task faces challenges due to the presence of different types of information, including neighboring entities, multi-modal attributes, and entity types. Directly incorporating the above information (e.g., concatenation or attention) can lead to an unaligned information space. To address these challenges, we propose a novel MMEA transformer, called MoAlign, that hierarchically introduces neighbor features, multi-modal attributes, and entity types to enhance the alignment task. Taking advantage of the transformer's ability to better integrate multiple information, we design a hierarchical modifiable self-attention block in a transformer encoder to preserve the unique semantics of different information. Furthermore, we design two entity-type prefix injection methods to integrate entity-type information using type prefixes, which help to restrict the global information of entities not present in the MMKGs. Our extensive experiments on benchmark datasets demonstrate that our approach outperforms strong competitors and achieves excellent entity alignment performance.