# VisionKG: Unleashing the Power of Visual Datasets via Knowledge Graph

arXiv: https://arxiv.org/abs/2309.13610 · HTML: https://mlsanity.com/paper/2309.13610/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 66 references that are in the map.
- First posted on arXiv 24 Sep 2023.

## Builds on (most cited first, 27 of 27)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020)
- [SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptation](https://mlsanity.com/paper/2206.08367/) (2022)
- [Simple multi-dataset detection](https://mlsanity.com/paper/2102.13086/) (2021)
- [MSeg: A Composite Dataset for Multi-domain Semantic Segmentation](https://mlsanity.com/paper/2112.13762/) (2021)
- [Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspective](https://mlsanity.com/paper/2112.06409/) (2021)
- [UA-DETRAC: A New Benchmark and Protocol for Multi-Object Detection and Tracking](https://mlsanity.com/paper/1511.04136/) (2015)
- [Towards Universal Object Detection by Domain Attention](https://mlsanity.com/paper/1904.04402/) (2019)
- [Semantic Relation Reasoning for Shot-Stable Few-Shot Object Detection](https://mlsanity.com/paper/2103.01903/) (2021)
- [SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognition](https://mlsanity.com/paper/1908.02660/) (2019)
- [A Data Quality-Driven View of MLOps](https://mlsanity.com/paper/2102.07750/) (2021)
- [Do We Need More Training Data?](https://mlsanity.com/paper/1503.01508/) (2015)
- [SDM-RDFizer: An RML Interpreter for the Efficient Creation of RDF Knowledge Graphs](https://mlsanity.com/paper/2008.07176/) (2020)
- [Deep Lake: a Lakehouse for Deep Learning](https://mlsanity.com/paper/2209.10785/) (2022)
- [OTPose: Occlusion-Aware Transformer for Pose Estimation in Sparsely-Labeled Videos](https://mlsanity.com/paper/2207.09725/) (2022)
- [Learning Visual Models using a Knowledge Graph as a Trainer](https://mlsanity.com/paper/2102.08747/) (2021)
- [Linking ImageNet WordNet Synsets with Wikidata](https://mlsanity.com/paper/1803.04349/) (2018)
- [Fantastic Data and How to Query Them](https://mlsanity.com/paper/2201.05026/) (2022)

## Built on by (most cited first, 5 of 5)

- [VHAKG: A Multi-modal Knowledge Graph Based on Synchronized Multi-view Videos of Daily Activities](https://mlsanity.com/paper/2408.14895/) (2024)
- [Segment Any Vehicle: Semantic and Visual Context Driven SAM and A Benchmark](https://mlsanity.com/paper/2508.04260/) (2025)
- [Cooperative Students: Navigating Unsupervised Domain Adaptation in Nighttime Object Detection](https://mlsanity.com/paper/2404.01988/) (2024)
- [Large Language Models as a Tool for Mining Object Knowledge](https://mlsanity.com/paper/2410.12959/) (2024)
- [Collaborative Perceiver: Elevating Vision-based 3D Object Detection via Local Density-Aware Spatial Occupancy](https://mlsanity.com/paper/2507.21358/) (2025)

## Abstract (arXiv)

The availability of vast amounts of visual data with heterogeneous features is a key factor for developing, testing, and benchmarking of new computer vision (CV) algorithms and architectures. Most visual datasets are created and curated for specific tasks or with limited image data distribution for very specific situations, and there is no unified approach to manage and access them across diverse sources, tasks, and taxonomies. This not only creates unnecessary overheads when building robust visual recognition systems, but also introduces biases into learning systems and limits the capabilities of data-centric AI. To address these problems, we propose the Vision Knowledge Graph (VisionKG), a novel resource that interlinks, organizes and manages visual datasets via knowledge graphs and Semantic Web technologies. It can serve as a unified framework facilitating simple access and querying of state-of-the-art visual datasets, regardless of their heterogeneous formats and taxonomies. One of the key differences between our approach and existing methods is that ours is knowledge-based rather than metadatabased. It enhances the enrichment of the semantics at both image and instance levels and offers various data retrieval and exploratory services via SPARQL. VisionKG currently contains 519 million RDF triples that describe approximately 40 million entities, and are accessible at https://vision.semkg.org and through APIs. With the integration of 30 datasets and four popular CV tasks, we demonstrate its usefulness across various scenarios when working with CV pipelines.