# Recognizing Unseen Objects via Multimodal Intensive Knowledge Graph Propagation

arXiv: https://arxiv.org/abs/2306.08487 · HTML: https://mlsanity.com/paper/2306.08487/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 55 references that are in the map.
- First posted on arXiv 14 Jun 2023.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks](https://mlsanity.com/paper/1905.07953/) (2019)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [Deep Convolutional Networks on Graph-Structured Data](https://mlsanity.com/paper/1506.05163/) (2015)
- [Zero-Shot Learning Through Cross-Modal Transfer](https://mlsanity.com/paper/1301.3666/) (2013)
- [Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs](https://mlsanity.com/paper/1803.08035/) (2018)
- [Zero-Shot Learning by Convex Combination of Semantic Embeddings](https://mlsanity.com/paper/1312.5650/) (2013)
- [Synthesized Classifiers for Zero-Shot Learning](https://mlsanity.com/paper/1603.00550/) (2016)
- [Rethinking Knowledge Graph Propagation for Zero-Shot Learning](https://mlsanity.com/paper/1805.11724/) (2018)
- [Zero-Shot Visual Recognition using Semantics-Preserving Adversarial Embedding Networks](https://mlsanity.com/paper/1712.01928/) (2017)
- [OntoZSL: Ontology-enhanced Zero-shot Learning](https://mlsanity.com/paper/2102.07339/) (2021)
- [Predicting Visual Exemplars of Unseen Classes for Zero-Shot Learning](https://mlsanity.com/paper/1605.08151/) (2016)
- [Attribute Propagation Network for Graph Zero-shot Learning](https://mlsanity.com/paper/2009.11816/) (2020)
- [VGSE: Visually-Grounded Semantic Embeddings for Zero-Shot Learning](https://mlsanity.com/paper/2203.10444/) (2022)
- [Zero-shot Node Classification with Decomposed Graph Prototype Network](https://mlsanity.com/paper/2106.08022/) (2021)
- [Learning the Implicit Semantic Representation on Graph-Structured Data](https://mlsanity.com/paper/2101.06471/) (2021)
- [Semi-supervised Vocabulary-informed Learning](https://mlsanity.com/paper/1604.07093/) (2016)
- [Disentangled Ontology Embedding for Zero-shot Learning](https://mlsanity.com/paper/2206.03739/) (2022)
- [Disentangled Action Recognition with Knowledge Bases](https://mlsanity.com/paper/2207.01708/) (2022)
- [Estimating Early Fundraising Performance of Innovations via Graph-based Market Environment Model](https://mlsanity.com/paper/1912.06767/) (2019)

## Built on by (most cited first, 1 of 1)

- [Dental Severity Assessment through Few-shot Learning and SBERT Fine-tuning](https://mlsanity.com/paper/2402.15755/) (2024)

## Abstract (arXiv)

Zero-Shot Learning (ZSL), which aims at automatically recognizing unseen objects, is a promising learning paradigm to understand new real-world knowledge for machines continuously. Recently, the Knowledge Graph (KG) has been proven as an effective scheme for handling the zero-shot task with large-scale and non-attribute data. Prior studies always embed relationships of seen and unseen objects into visual information from existing knowledge graphs to promote the cognitive ability of the unseen data. Actually, real-world knowledge is naturally formed by multimodal facts. Compared with ordinary structural knowledge from a graph perspective, multimodal KG can provide cognitive systems with fine-grained knowledge. For example, the text description and visual content can depict more critical details of a fact than only depending on knowledge triplets. Unfortunately, this multimodal fine-grained knowledge is largely unexploited due to the bottleneck of feature alignment between different modalities. To that end, we propose a multimodal intensive ZSL framework that matches regions of images with corresponding semantic embeddings via a designed dense attention module and self-calibration loss. It makes the semantic transfer process of our ZSL framework learns more differentiated knowledge between entities. Our model also gets rid of the performance limitation of only using rough global features. We conduct extensive experiments and evaluate our model on large-scale real-world data. The experimental results clearly demonstrate the effectiveness of the proposed model in standard zero-shot classification tasks.