# Detection Hub: Unifying Object Detection Datasets via Query Adaptation on Language Embedding

arXiv: https://arxiv.org/abs/2206.03484 · HTML: https://mlsanity.com/paper/2206.03484/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 52 references that are in the map.
- First posted on arXiv 7 Jun 2022.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [DOTA: A Large-scale Dataset for Object Detection in Aerial Images](https://mlsanity.com/paper/1711.10398/) (2017)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [Cross-Domain Weakly-Supervised Object Detection through Progressive Domain Adaptation](https://mlsanity.com/paper/1803.11365/) (2018)
- [BEVT: BERT Pretraining of Video Transformers](https://mlsanity.com/paper/2112.01529/) (2021)
- [Simple multi-dataset detection](https://mlsanity.com/paper/2102.13086/) (2021)
- [MSeg: A Composite Dataset for Multi-domain Semantic Segmentation](https://mlsanity.com/paper/2112.13762/) (2021)

## Built on by (most cited first, 25 of 25)

- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [Detecting Everything in the Open World: Towards Universal Object Detection](https://mlsanity.com/paper/2303.11749/) (2023)
- [X-Paste: Revisiting Scalable Copy-Paste for Instance Segmentation using CLIP and StableDiffusion](https://mlsanity.com/paper/2212.03863/) (2022)
- [DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model](https://mlsanity.com/paper/2306.01736/) (2023)
- [Multi-Prompt Alignment for Multi-Source Unsupervised Domain Adaptation](https://mlsanity.com/paper/2209.15210/) (2022)
- [UniDet3D: Multi-dataset Indoor 3D Object Detection](https://mlsanity.com/paper/2409.04234/) (2024)
- [SEGIC: Unleashing the Emergent Correspondence for In-Context Segmentation](https://mlsanity.com/paper/2311.14671/) (2023)
- [Learning from Rich Semantics and Coarse Locations for Long-tailed Object Detection](https://mlsanity.com/paper/2310.12152/) (2023)
- [FlexAvatar: Learning Complete 3D Head Avatars with Partial Supervision](https://mlsanity.com/paper/2512.15599/) (2025)
- [Plain-Det: A Plain Multi-Dataset Object Detector](https://mlsanity.com/paper/2407.10083/) (2024)
- [One for All: Multi-Domain Joint Training for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/2411.01584/) (2024)
- [INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuning](https://mlsanity.com/paper/2412.03565/) (2024)
- [Universal Object Detection with Large Vision Model](https://mlsanity.com/paper/2212.09408/) (2022)
- [Language-aware Multiple Datasets Detection Pretraining for DETRs](https://mlsanity.com/paper/2304.03580/) (2023)
- [Bridging Annotation Gaps: Transferring Labels to Align Object Detection Datasets](https://mlsanity.com/paper/2506.04737/) (2025)
- [TMT-VIS: Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation](https://mlsanity.com/paper/2312.06630/) (2023)
- [Infinite 3D Landmarks: Improving Continuous 2D Facial Landmark Detection](https://mlsanity.com/paper/2405.20117/) (2024)
- [Automated Label Unification for Multi-Dataset Semantic Segmentation with GNNs](https://mlsanity.com/paper/2407.10534/) (2024)
- [Comprehensive Multi-Modal Prototypes are Simple and Effective Classifiers for Vast-Vocabulary Object Detection](https://mlsanity.com/paper/2412.17800/) (2024)
- [The COTe score: A decomposable framework for evaluating Document Layout Analysis models](https://mlsanity.com/paper/2603.12718/) (2026)
- [Hi-OPD: Hierarchy-Aware Open-Prompt Detection for Remote Sensing Images](https://mlsanity.com/paper/2609.25584/) (2026)
- [Online Pseudo-Label Unified Object Detection for Multiple Datasets Training](https://mlsanity.com/paper/2410.15569/) (2024)
- [Improving Detection of Person Class Using Dense Pooling](https://mlsanity.com/paper/2410.20966/) (2024)
- [Multi-Domain Learning with Global Expert Mapping](https://mlsanity.com/paper/2604.18842/) (2026)
- [HMR-Net: Hierarchical Modular Routing for Cross-Domain Object Detection in Aerial Images](https://mlsanity.com/paper/2604.18866/) (2026)

## Abstract (arXiv)

Combining multiple datasets enables performance boost on many computer vision tasks. But similar trend has not been witnessed in object detection when combining multiple datasets due to two inconsistencies among detection datasets: taxonomy difference and domain gap. In this paper, we address these challenges by a new design (named Detection Hub) that is dataset-aware and category-aligned. It not only mitigates the dataset inconsistency but also provides coherent guidance for the detector to learn across multiple datasets. In particular, the dataset-aware design is achieved by learning a dataset embedding that is used to adapt object queries as well as convolutional kernels in detection heads. The categories across datasets are semantically aligned into a unified space by replacing one-hot category representations with word embedding and leveraging the semantic coherence of language embedding. Detection Hub fulfills the benefits of large data on object detection. Experiments demonstrate that joint training on multiple datasets achieves significant performance gains over training on each dataset alone. Detection Hub further achieves SoTA performance on UODB benchmark with wide variety of datasets.