# Object Detection with a Unified Label Space from Multiple Datasets

arXiv: https://arxiv.org/abs/2008.06614 · HTML: https://mlsanity.com/paper/2008.06614/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2020.

## Built on by (most cited first, 30 of 65)

- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Simple multi-dataset detection](https://mlsanity.com/paper/2102.13086/) (2021)
- [Exploiting Unlabeled Data with Vision and Language Models for Object Detection](https://mlsanity.com/paper/2207.08954/) (2022)
- [Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training](https://mlsanity.com/paper/2308.09718/) (2023)
- [Detecting Everything in the Open World: Towards Universal Object Detection](https://mlsanity.com/paper/2303.11749/) (2023)
- [Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection](https://mlsanity.com/paper/2303.06880/) (2023)
- [INTERN: A New Learning Paradigm Towards General Vision](https://mlsanity.com/paper/2111.08687/) (2021)
- [BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-training](https://mlsanity.com/paper/2203.13249/) (2022)
- [GAIA: A Transfer Learning System of Object Detection that Fits Your Needs](https://mlsanity.com/paper/2106.11346/) (2021)
- [AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving](https://mlsanity.com/paper/2403.17373/) (2024)
- [Detection Hub: Unifying Object Detection Datasets via Query Adaptation on Language Embedding](https://mlsanity.com/paper/2206.03484/) (2022)
- [DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasets](https://mlsanity.com/paper/2311.04894/) (2023)
- [ROADWork: A Dataset and Benchmark for Learning to Recognize, Observe, Analyze and Drive Through Work Zones](https://mlsanity.com/paper/2406.07661/) (2024)
- [SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detection](https://mlsanity.com/paper/2412.20665/) (2024)
- [UniDet3D: Multi-dataset Indoor 3D Object Detection](https://mlsanity.com/paper/2409.04234/) (2024)
- [Diffusion-Based Hierarchical Multi-Label Object Detection to Analyze Panoramic Dental X-rays](https://mlsanity.com/paper/2303.06500/) (2023)
- [Spatial Consistency Loss for Training Multi-Label Classifiers from Single-Label Annotations](https://mlsanity.com/paper/2203.06127/) (2022)
- [OmniLabel: A Challenging Benchmark for Language-Based Object Detection](https://mlsanity.com/paper/2304.11463/) (2023)
- [ScaleDet: A Scalable Multi-Dataset Object Detector](https://mlsanity.com/paper/2306.04849/) (2023)
- [PatchRefiner: Leveraging Synthetic Data for Real-Domain High-Resolution Monocular Metric Depth Estimation](https://mlsanity.com/paper/2406.06679/) (2024)
- [GM-DF: Generalized Multi-Scenario Deepfake Detection](https://mlsanity.com/paper/2406.20078/) (2024)
- [LMSeg: Language-guided Multi-dataset Segmentation](https://mlsanity.com/paper/2302.13495/) (2023)
- [Multi-domain semantic segmentation with overlapping labels](https://mlsanity.com/paper/2108.11224/) (2021)
- [Learning Semantic Segmentation from Multiple Datasets with Label Shifts](https://mlsanity.com/paper/2202.14030/) (2022)
- [MDT3D: Multi-Dataset Training for LiDAR 3D Object Detection Generalization](https://mlsanity.com/paper/2308.01000/) (2023)
- [Unified Visual Relationship Detection with Vision and Language Models](https://mlsanity.com/paper/2303.08998/) (2023)
- [USB: Universal-Scale Object Detection Benchmark](https://mlsanity.com/paper/2103.14027/) (2021)
- [Plain-Det: A Plain Multi-Dataset Object Detector](https://mlsanity.com/paper/2407.10083/) (2024)
- [COLA: COarse-LAbel multi-source LiDAR semantic segmentation for autonomous driving](https://mlsanity.com/paper/2311.03017/) (2023)
- [One for All: Multi-Domain Joint Training for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/2411.01584/) (2024)

## Abstract (arXiv)

Given multiple datasets with different label spaces, the goal of this work is to train a single object detector predicting over the union of all the label spaces. The practical benefits of such an object detector are obvious and significant application-relevant categories can be picked and merged form arbitrary existing datasets. However, naive merging of datasets is not possible in this case, due to inconsistent object annotations. Consider an object category like faces that is annotated in one dataset, but is not annotated in another dataset, although the object itself appears in the latter images. Some categories, like face here, would thus be considered foreground in one dataset, but background in another. To address this challenge, we design a framework which works with such partial annotations, and we exploit a pseudo labeling approach that we adapt for our specific case. We propose loss functions that carefully integrate partial but correct annotations with complementary but noisy pseudo labels. Evaluation in the proposed novel setting requires full annotation on the test set. We collect the required annotations and define a new challenging experimental setup for this task based one existing public datasets. We show improved performances compared to competitive baselines and appropriate adaptations of existing work.