# End-to-End Entity Classification on Multimodal Knowledge Graphs

arXiv: https://arxiv.org/abs/2003.12383 · HTML: https://mlsanity.com/paper/2003.12383/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2020.

## Built on by (most cited first, 12 of 12)

- [A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodal](https://mlsanity.com/paper/2212.05767/) (2022)
- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [Contrastive Language-Image Pre-Training with Knowledge Graphs](https://mlsanity.com/paper/2210.08901/) (2022)
- [Query Embedding on Hyper-relational Knowledge Graphs](https://mlsanity.com/paper/2106.08166/) (2021)
- [Cross-modal Retrieval for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2401.05736/) (2024)
- [Learning Visual Models using a Knowledge Graph as a Trainer](https://mlsanity.com/paper/2102.08747/) (2021)
- [Entity Type Prediction Leveraging Graph Walks and Entity Descriptions](https://mlsanity.com/paper/2207.14094/) (2022)
- [Towards Semantically Enriched Embeddings for Knowledge Graph Completion](https://mlsanity.com/paper/2308.00081/) (2023)
- [Domain Adaptation for Large-Vocabulary Object Detectors](https://mlsanity.com/paper/2401.06969/) (2024)
- [ReaLitE: Enrichment of Relation Embeddings in Knowledge Graphs using Numeric Literals](https://mlsanity.com/paper/2504.00852/) (2025)
- [A Universal Knowledge Model and Cognitive Architecture for Prototyping AGI](https://mlsanity.com/paper/2401.06256/) (2024)
- [Relations Are Channels: Knowledge Graph Embedding via Kraus Decompositions](https://mlsanity.com/paper/2605.10317/) (2026)

## Abstract (arXiv)

End-to-end multimodal learning on knowledge graphs has been left largely unaddressed. Instead, most end-to-end models such as message passing networks learn solely from the relational information encoded in graphs' structure: raw values, or literals, are either omitted completely or are stripped from their values and treated as regular nodes. In either case we lose potentially relevant information which could have otherwise been exploited by our learning methods. To avoid this, we must treat literals and non-literals as separate cases. We must also address each modality separately and accordingly: numbers, texts, images, geometries, et cetera. We propose a multimodal message passing network which not only learns end-to-end from the structure of graphs, but also from their possibly divers set of multimodal node features. Our model uses dedicated (neural) encoders to naturally learn embeddings for node features belonging to five different types of modalities, including images and geometries, which are projected into a joint representation space together with their relational information. We demonstrate our model on a node classification task, and evaluate the effect that each modality has on the overall performance. Our result supports our hypothesis that including information from multiple modalities can help our models obtain a better overall performance.