# Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillation

arXiv: https://arxiv.org/abs/2203.10593 · HTML: https://mlsanity.com/paper/2203.10593/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 35 of its 47 references that are in the map.
- First posted on arXiv 20 Mar 2022.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection](https://mlsanity.com/paper/1912.02424/) (2019)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://mlsanity.com/paper/2006.04388/) (2020)
- [Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers](https://mlsanity.com/paper/2004.00849/) (2020)
- [TOOD: Task-aligned One-stage Object Detection](https://mlsanity.com/paper/2108.07755/) (2021)
- [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018)
- [Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs](https://mlsanity.com/paper/1803.08035/) (2018)
- [Probabilistic Anchor Assignment with IoU Prediction for Object Detection](https://mlsanity.com/paper/2007.08103/) (2020)
- [Zero-Shot Learning by Convex Combination of Semantic Embeddings](https://mlsanity.com/paper/1312.5650/) (2013)
- [Generalized Focal Loss V2: Learning Reliable Localization Quality Estimation for Dense Object Detection](https://mlsanity.com/paper/2011.12885/) (2020)
- [Dont Even Look Once: Synthesizing Features for Zero-Shot Detection](https://mlsanity.com/paper/1911.07933/) (2019)

## Built on by (most cited first, 30 of 68)

- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2303.05892/) (2023)
- [Detecting Everything in the Open World: Towards Universal Object Detection](https://mlsanity.com/paper/2303.11749/) (2023)
- [HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models](https://mlsanity.com/paper/2303.15786/) (2023)
- [A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future](https://mlsanity.com/paper/2307.09220/) (2023)
- [CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection](https://mlsanity.com/paper/2310.02960/) (2023)
- [LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors](https://mlsanity.com/paper/2402.04630/) (2024)
- [Open-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfer](https://mlsanity.com/paper/2207.01887/) (2022)
- [Distilling Large Vision-Language Model with Out-of-Distribution Generalizability](https://mlsanity.com/paper/2307.03135/) (2023)
- [Defense-Prefix for Preventing Typographic Attacks on CLIP](https://mlsanity.com/paper/2304.04512/) (2023)
- [Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection](https://mlsanity.com/paper/2406.00510/) (2024)
- [Open Vocabulary Object Detection with Proposal Mining and Prediction Equalization](https://mlsanity.com/paper/2206.11134/) (2022)
- [Open Vocabulary Monocular 3D Object Detection](https://mlsanity.com/paper/2411.16833/) (2024)
- [Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detection](https://mlsanity.com/paper/2302.00268/) (2023)
- [Prompt-Guided Zero-Shot Anomaly Action Recognition using Pretrained Deep Skeleton Features](https://mlsanity.com/paper/2303.15167/) (2023)
- [From Text to Pixels: A Context-Aware Semantic Synergy Solution for Infrared and Visible Image Fusion](https://mlsanity.com/paper/2401.00421/) (2023)
- [OV-Uni3DETR: Towards Unified Open-Vocabulary 3D Object Detection via Cycle-Modality Propagation](https://mlsanity.com/paper/2403.19580/) (2024)
- [Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization](https://mlsanity.com/paper/2401.15914/) (2024)
- [Collaborative Novel Object Discovery and Box-Guided Cross-Modal Alignment for Open-Vocabulary 3D Object Detection](https://mlsanity.com/paper/2406.00830/) (2024)
- [ImOV3D: Learning Open-Vocabulary Point Clouds 3D Object Detection from Only 2D Images](https://mlsanity.com/paper/2410.24001/) (2024)
- [Mask-free OVIS: Open-Vocabulary Instance Segmentation without Manual Mask Annotations](https://mlsanity.com/paper/2303.16891/) (2023)
- [KD-DLGAN: Data Limited Image Generation via Knowledge Distillation](https://mlsanity.com/paper/2303.17158/) (2023)
- [GridCLIP: One-Stage Object Detection by Grid-Level CLIP Representation Learning](https://mlsanity.com/paper/2303.09252/) (2023)
- [Unified Embedding Alignment for Open-Vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2407.07427/) (2024)
- [Disentangle and Remerge: Interventional Knowledge Distillation for Few-Shot Object Detection from A Conditional Causal Perspective](https://mlsanity.com/paper/2208.12681/) (2022)
- [Open World Object Detection: A Survey](https://mlsanity.com/paper/2410.11301/) (2024)
- [Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2503.16707/) (2025)
- [Refining CLIP's Spatial Awareness: A Visual-Centric Perspective](https://mlsanity.com/paper/2504.02328/) (2025)
- [Learning from Rich Semantics and Coarse Locations for Long-tailed Object Detection](https://mlsanity.com/paper/2310.12152/) (2023)

## Abstract (arXiv)

Open-vocabulary object detection aims to detect novel object categories beyond the training set. The advanced open-vocabulary two-stage detectors employ instance-level visual-to-visual knowledge distillation to align the visual space of the detector with the semantic space of the Pre-trained Visual-Language Model (PVLM). However, in the more efficient one-stage detector, the absence of class-agnostic object proposals hinders the knowledge distillation on unseen objects, leading to severe performance degradation. In this paper, we propose a hierarchical visual-language knowledge distillation method, i.e., HierKD, for open-vocabulary one-stage detection. Specifically, a global-level knowledge distillation is explored to transfer the knowledge of unseen categories from the PVLM to the detector. Moreover, we combine the proposed global-level knowledge distillation and the common instance-level knowledge distillation to learn the knowledge of seen and unseen categories simultaneously. Extensive experiments on MS-COCO show that our method significantly surpasses the previous best one-stage detector with 11.9% and 6.7% AP50 gains under the zero-shot detection and generalized zero-shot detection settings, and reduces the AP50 performance gap from 14% to 7.3% compared to the best two-stage detector.