# The More You Know: Using Knowledge Graphs for Image Classification

arXiv: https://arxiv.org/abs/1612.04844 · HTML: https://mlsanity.com/paper/1612.04844/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2016.

## Built on by (most cited first, 30 of 63)

- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge](https://mlsanity.com/paper/2209.07118/) (2022)
- [Hierarchical Object-to-Zone Graph for Object Navigation](https://mlsanity.com/paper/2109.02066/) (2021)
- [TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priors](https://mlsanity.com/paper/2207.10761/) (2022)
- [Dual Graph Convolutional Networks with Transformer and Curriculum Learning for Image Captioning](https://mlsanity.com/paper/2108.02366/) (2021)
- [Act Like a Radiologist: Towards Reliable Multi-view Correspondence Reasoning for Mammogram Mass Detection](https://mlsanity.com/paper/2105.10160/) (2021)
- [Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review](https://mlsanity.com/paper/2303.06471/) (2023)
- [Network Representation Learning: From Preprocessing, Feature Extraction to Node Embedding](https://mlsanity.com/paper/2110.07582/) (2021)
- [The Pursuit of Knowledge: Discovering and Localizing Novel Categories using Dual Memory](https://mlsanity.com/paper/2105.01652/) (2021)
- [GraphFM: Graph Factorization Machines for Feature Interaction Modeling](https://mlsanity.com/paper/2105.11866/) (2021)
- [Taxonomy-Aware Evaluation of Vision-Language Models](https://mlsanity.com/paper/2504.05457/) (2025)
- [Faster-LTN: a neuro-symbolic, end-to-end object detection architecture](https://mlsanity.com/paper/2107.01877/) (2021)
- [Applications of knowledge graphs for food science and industry](https://mlsanity.com/paper/2107.05869/) (2021)
- [Sample-Efficient Learning of Novel Visual Concepts](https://mlsanity.com/paper/2306.09482/) (2023)
- [Survey of Image Based Graph Neural Networks](https://mlsanity.com/paper/2106.06307/) (2021)
- [A Survey on Visual Transfer Learning using Knowledge Graphs](https://mlsanity.com/paper/2201.11794/) (2022)
- [Duality-Induced Regularizer for Semantic Matching Knowledge Graph Embeddings](https://mlsanity.com/paper/2203.12949/) (2022)
- [DyGCN: Dynamic Graph Embedding with Graph Convolutional Network](https://mlsanity.com/paper/2104.02962/) (2021)
- [Increasing Coverage and Precision of Textual Information in Multilingual Knowledge Graphs](https://mlsanity.com/paper/2311.15781/) (2023)
- [GMLP: Building Scalable and Flexible Graph Neural Networks with Feature-Message Passing](https://mlsanity.com/paper/2104.09880/) (2021)
- [Magic Layouts: Structural Prior for Component Detection in User Interface Designs](https://mlsanity.com/paper/2106.07615/) (2021)
- [KTN: Knowledge Transfer Network for Learning Multi-person 2D-3D Correspondences](https://mlsanity.com/paper/2206.10090/) (2022)
- [Capturing Knowledge Graphs and Rules with Octagon Embeddings](https://mlsanity.com/paper/2401.16270/) (2024)
- [Addressing materials' microstructure diversity using transfer learning](https://mlsanity.com/paper/2107.13841/) (2021)
- [Ontology-based n-ball Concept Embeddings Informing Few-shot Image Classification](https://mlsanity.com/paper/2109.09063/) (2021)
- [DRAG: Dynamic Region-Aware GCN for Privacy-Leaking Image Detection](https://mlsanity.com/paper/2203.09121/) (2022)
- [Brainish: Formalizing A Multimodal Language for Intelligence and Consciousness](https://mlsanity.com/paper/2205.00001/) (2022)
- [ER: Equivariance Regularizer for Knowledge Graph Completion](https://mlsanity.com/paper/2206.12142/) (2022)
- [Content-based Graph Privacy Advisor](https://mlsanity.com/paper/2210.11169/) (2022)
- [Graph Neural Networks in Computer Vision -- Architectures, Datasets and Common Approaches](https://mlsanity.com/paper/2212.10207/) (2022)

## Abstract (arXiv)

One characteristic that sets humans apart from modern learning-based computer vision algorithms is the ability to acquire knowledge about the world and use that knowledge to reason about the visual world. Humans can learn about the characteristics of objects and the relationships that occur between them to learn a large variety of visual concepts, often with few examples. This paper investigates the use of structured prior knowledge in the form of knowledge graphs and shows that using this knowledge improves performance on image classification. We build on recent work on end-to-end learning on graphs, introducing the Graph Search Neural Network as a way of efficiently incorporating large knowledge graphs into a vision classification pipeline. We show in a number of experiments that our method outperforms standard neural network baselines for multi-label classification.