# Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection

arXiv: https://arxiv.org/abs/2303.05892 · HTML: https://mlsanity.com/paper/2303.05892/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 60 references that are in the map.
- First posted on arXiv 10 Mar 2023.

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model](https://mlsanity.com/paper/2203.14940/) (2022)
- [Simple Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2205.06230/) (2022)
- [Open-Vocabulary DETR with Conditional Matching](https://mlsanity.com/paper/2203.11876/) (2022)
- [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018)
- [Distilling Object Detectors with Fine-grained Feature Imitation](https://mlsanity.com/paper/1906.03609/) (2019)
- [Weakly Supervised Deep Detection Networks](https://mlsanity.com/paper/1511.02853/) (2015)
- [DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection](https://mlsanity.com/paper/2209.09407/) (2022)
- [Focal and Global Knowledge Distillation for Detectors](https://mlsanity.com/paper/2111.11837/) (2021)
- [GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection](https://mlsanity.com/paper/2203.13954/) (2022)
- [General Instance Distillation for Object Detection](https://mlsanity.com/paper/2103.02340/) (2021)
- [Aligning Pretraining for Detection via Object-Level Contrastive Learning](https://mlsanity.com/paper/2106.02637/) (2021)

## Built on by (most cited first, 30 of 63)

- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance](https://mlsanity.com/paper/2312.10671/) (2023)
- [CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction](https://mlsanity.com/paper/2310.01403/) (2023)
- [A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future](https://mlsanity.com/paper/2307.09220/) (2023)
- [DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLM](https://mlsanity.com/paper/2403.12488/) (2024)
- [ViTamin: Designing Scalable Vision Models in the Vision-Language Era](https://mlsanity.com/paper/2404.02132/) (2024)
- [DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception](https://mlsanity.com/paper/2505.04410/) (2025)
- [Tokenize Anything via Prompting](https://mlsanity.com/paper/2312.09128/) (2023)
- [Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2406.02548/) (2024)
- [Generative Region-Language Pretraining for Open-Ended Object Detection](https://mlsanity.com/paper/2403.10191/) (2024)
- [InstaGen: Enhancing Object Detection by Training on Synthetic Dataset](https://mlsanity.com/paper/2402.05937/) (2024)
- [Taming Self-Training for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2308.06412/) (2023)
- [Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection](https://mlsanity.com/paper/2406.00510/) (2024)
- [OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-World Unknown Objects Supervision](https://mlsanity.com/paper/2405.17913/) (2024)
- [Text Augmented Spatial-aware Zero-shot Referring Image Segmentation](https://mlsanity.com/paper/2310.18049/) (2023)
- [CLIM: Contrastive Language-Image Mosaic for Region Representation](https://mlsanity.com/paper/2312.11376/) (2023)
- [Retrieval-Augmented Open-Vocabulary Object Detection](https://mlsanity.com/paper/2404.05687/) (2024)
- [LaMI-DETR: Open-Vocabulary Detection with Language Model Instruction](https://mlsanity.com/paper/2407.11335/) (2024)
- [ImOV3D: Learning Open-Vocabulary Point Clouds 3D Object Detection from Only 2D Images](https://mlsanity.com/paper/2410.24001/) (2024)
- [WeDetect: Fast Open-Vocabulary Object Detection as Retrieval](https://mlsanity.com/paper/2512.12309/) (2025)
- [DSRC: Learning Density-insensitive and Semantic-aware Collaborative Representation against Corruptions](https://mlsanity.com/paper/2412.10739/) (2024)
- [Refining CLIP's Spatial Awareness: A Visual-Centric Perspective](https://mlsanity.com/paper/2504.02328/) (2025)
- [DiPEx: Dispersing Prompt Expansion for Class-Agnostic Object Detection](https://mlsanity.com/paper/2406.14924/) (2024)
- [MarvelOVD: Marrying Object Recognition and Vision-Language Models for Robust Open-Vocabulary Object Detection](https://mlsanity.com/paper/2407.21465/) (2024)
- [Vision-Language Model for Object Detection and Segmentation: A Review and Evaluation](https://mlsanity.com/paper/2504.09480/) (2025)
- [Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception](https://mlsanity.com/paper/2508.11256/) (2025)
- [A Hierarchical Semantic Distillation Framework for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2503.10152/) (2025)
- [Region-centric Image-Language Pretraining for Open-Vocabulary Detection](https://mlsanity.com/paper/2310.00161/) (2023)
- [Details Matter for Indoor Open-vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2507.23134/) (2025)

## Abstract (arXiv)

Open-vocabulary object detection aims to provide object detectors trained on a fixed set of object categories with the generalizability to detect objects described by arbitrary text queries. Previous methods adopt knowledge distillation to extract knowledge from Pretrained Vision-and-Language Models (PVLMs) and transfer it to detectors. However, due to the non-adaptive proposal cropping and single-level feature mimicking processes, they suffer from information destruction during knowledge extraction and inefficient knowledge transfer. To remedy these limitations, we propose an Object-Aware Distillation Pyramid (OADP) framework, including an Object-Aware Knowledge Extraction (OAKE) module and a Distillation Pyramid (DP) mechanism. When extracting object knowledge from PVLMs, the former adaptively transforms object proposals and adopts object-aware mask attention to obtain precise and complete knowledge of objects. The latter introduces global and block distillation for more comprehensive knowledge transfer to compensate for the missing relation information in object distillation. Extensive experiments show that our method achieves significant improvement compared to current methods. Especially on the MS-COCO dataset, our OADP framework reaches 35.6 mAPN50, surpassing the current state-of-the-art method by 3.3 mAPN50. Code is released at https://github.com/LutingWang/OADP.