# Context-driven Visual Object Recognition based on Knowledge Graphs

arXiv: https://arxiv.org/abs/2210.11233 · HTML: https://mlsanity.com/paper/2210.11233/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 56 references that are in the map.
- First posted on arXiv 20 Oct 2022.

## Builds on (most cited first, 29 of 29)

- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet](https://mlsanity.com/paper/1904.00760/) (2019)
- [Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks](https://mlsanity.com/paper/1810.12348/) (2018)
- [Graph-Structured Representations for Visual Question Answering](https://mlsanity.com/paper/1609.05600/) (2016)
- [Learning Graph Embeddings for Compositional Zero-shot Learning](https://mlsanity.com/paper/2102.01987/) (2021)
- [Don't Judge an Object by Its Context: Learning to Overcome Contextual Bias](https://mlsanity.com/paper/2001.03152/) (2020)
- [Structure Inference Net: Object Detection Using Scene-Level Context and Instance-Level Relationships](https://mlsanity.com/paper/1807.00119/) (2018)
- [Iterative Visual Reasoning Beyond Convolutions](https://mlsanity.com/paper/1803.11189/) (2018)
- [Putting visual object recognition in context](https://mlsanity.com/paper/1911.07349/) (2019)
- [CoKE: Contextualized Knowledge Graph Embedding](https://mlsanity.com/paper/1911.02168/) (2019)
- [Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition](https://mlsanity.com/paper/1511.04196/) (2015)
- [Learning Structured Inference Neural Networks with Label Relations](https://mlsanity.com/paper/1511.05616/) (2015)
- [DOLORES: Deep Contextualized Knowledge Graph Embeddings](https://mlsanity.com/paper/1811.00147/) (2018)
- [LightCAKE: A Lightweight Framework for Context-Aware Knowledge Graph Embedding](https://mlsanity.com/paper/2102.10826/) (2021)
- [Learning Visual Models using a Knowledge Graph as a Trainer](https://mlsanity.com/paper/2102.08747/) (2021)
- [Context-Aware Zero-Shot Learning for Object Recognition](https://mlsanity.com/paper/1904.12638/) (2019)
- [A Survey on Visual Transfer Learning using Knowledge Graphs](https://mlsanity.com/paper/2201.11794/) (2022)
- [Learning Scene Gist with Convolutional Neural Networks to Improve Object Recognition](https://mlsanity.com/paper/1803.01967/) (2018)
- [Lift-the-flap: what, where and when for context reasoning](https://mlsanity.com/paper/1902.00163/) (2019)

## Abstract (arXiv)

Current deep learning methods for object recognition are purely data-driven and require a large number of training samples to achieve good results. Due to their sole dependence on image data, these methods tend to fail when confronted with new environments where even small deviations occur. Human perception, however, has proven to be significantly more robust to such distribution shifts. It is assumed that their ability to deal with unknown scenarios is based on extensive incorporation of contextual knowledge. Context can be based either on object co-occurrences in a scene or on memory of experience. In accordance with the human visual cortex which uses context to form different object representations for a seen image, we propose an approach that enhances deep learning methods by using external contextual knowledge encoded in a knowledge graph. Therefore, we extract different contextual views from a generic knowledge graph, transform the views into vector space and infuse it into a DNN. We conduct a series of experiments to investigate the impact of different contextual views on the learned object representations for the same image dataset. The experimental results provide evidence that the contextual views influence the image representations in the DNN differently and therefore lead to different predictions for the same images. We also show that context helps to strengthen the robustness of object recognition models for out-of-distribution images, usually occurring in transfer learning tasks or real-world scenarios.