# Knowledge Distillation via Query Selection for Detection Transformer arXiv: https://arxiv.org/abs/2409.06443 ยท HTML: https://mlsanity.com/paper/2409.06443/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation. - Builds on 29 of its 46 references that are in the map. - First posted on arXiv 10 Sep 2024. ## Placed in - Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/) ## Builds on (most cited first, 29 of 29) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020) - [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020) - [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014) - [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019) - [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019) - [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022) - [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022) - [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021) - [DETRs with Collaborative Hybrid Assignments Training](https://mlsanity.com/paper/2211.12860/) (2022) - [Distilling Object Detectors with Fine-grained Feature Imitation](https://mlsanity.com/paper/1906.03609/) (2019) - [Focal and Global Knowledge Distillation for Detectors](https://mlsanity.com/paper/2111.11837/) (2021) - [Distilling Object Detectors via Decoupled Features](https://mlsanity.com/paper/2103.14475/) (2021) - [Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment](https://mlsanity.com/paper/2207.13085/) (2022) - [AdaMixer: A Fast-Converging Query-Based Object Detector](https://mlsanity.com/paper/2203.16507/) (2022) - [Instance-Conditional Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2110.12724/) (2021) - [Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-guided Feature Imitation](https://mlsanity.com/paper/2112.04840/) (2021) - [Distilling Object Detectors with Feature Richness](https://mlsanity.com/paper/2111.00674/) (2021) - [Distilling Object Detectors with Task Adaptive Regularization](https://mlsanity.com/paper/2006.13108/) (2020) - [DETRDistill: A Universal Knowledge Distillation Framework for DETR-families](https://mlsanity.com/paper/2211.10156/) (2022) - [Structural Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2211.13133/) (2022) - [HEAD: HEtero-Assists Distillation for Heterogeneous Object Detectors](https://mlsanity.com/paper/2207.05345/) (2022) - [Robust Table Structure Recognition with Dynamic Queries Enhanced Detection Transformer](https://mlsanity.com/paper/2303.11615/) (2023) - [Faster OreFSDet : A Lightweight and Effective Few-shot Object Detector for Ore Images](https://mlsanity.com/paper/2305.01183/) (2023) - [A Multi-Modal Transformer Network for Action Detection](https://mlsanity.com/paper/2305.19624/) (2023) ## Built on by (most cited first, 5 of 5) - [CLoCKDistill: Consistent Location-and-Context-aware Knowledge Distillation for DETRs](https://mlsanity.com/paper/2502.10683/) (2025) - [Architectural Insights into Knowledge Distillation for Object Detection: A Comprehensive Review](https://mlsanity.com/paper/2508.03317/) (2025) - [Bounded Behavioral Indistinguishability for Black-Box LLM Distillation](https://mlsanity.com/paper/2605.30448/) (2026) - [Enhanced Knowledge Distillation for Detection Transformer via Teacher Prediction Refinement](https://mlsanity.com/paper/2609.19964/) (2026) - [Match One, Learn with Graph: One-to-Graph Query Collaboration with Backward Sharing for Object Detection](https://mlsanity.com/paper/2609.26092/) (2026) ## Abstract (arXiv) Transformers have revolutionized the object detection landscape by introducing DETRs, acclaimed for their simplicity and efficacy. Despite their advantages, the substantial size of these models poses significant challenges for practical deployment, particularly in resource-constrained environments. This paper addresses the challenge of compressing DETR by leveraging knowledge distillation, a technique that holds promise for maintaining model performance while reducing size. A critical aspect of DETRs' performance is their reliance on queries to interpret object representations accurately. Traditional distillation methods often focus exclusively on positive queries, identified through bipartite matching, neglecting the rich information present in hard-negative queries. Our visual analysis indicates that hard-negative queries, focusing on foreground elements, are crucial for enhancing distillation outcomes. To this end, we introduce a novel Group Query Selection strategy, which diverges from traditional query selection in DETR distillation by segmenting queries based on their Generalized Intersection over Union (GIoU) with ground truth objects, thereby uncovering valuable hard-negative queries for distillation. Furthermore, we present the Knowledge Distillation via Query Selection for DETR (QSKD) framework, which incorporates Attention-Guided Feature Distillation (AGFD) and Local Alignment Prediction Distillation (LAPD). These components optimize the distillation process by focusing on the most informative aspects of the teacher model's intermediate features and output. Our comprehensive experimental evaluation of the MS-COCO dataset demonstrates the effectiveness of our approach, significantly improving average precision (AP) across various DETR architectures without incurring substantial computational costs. Specifically, the AP of Conditional DETR ResNet-18 increased from 35.8 to 39.9.