# Enhanced Knowledge Distillation for Detection Transformer via Teacher Prediction Refinement

arXiv: https://arxiv.org/abs/2609.19964 · HTML: https://mlsanity.com/paper/2609.19964/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 48 of its 56 references that are in the map.
- First posted on arXiv 17 Sep 2026.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 30 of 48)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [YOLOv12: Attention-Centric Real-Time Object Detectors](https://mlsanity.com/paper/2502.12524/) (2025)
- [Distilling Knowledge via Knowledge Review](https://mlsanity.com/paper/2104.09044/) (2021)
- [DETRs with Hybrid Matching](https://mlsanity.com/paper/2207.13080/) (2022)
- [Focal and Global Knowledge Distillation for Detectors](https://mlsanity.com/paper/2111.11837/) (2021)
- [TIDE: A General Toolbox for Identifying Object Detection Errors](https://mlsanity.com/paper/2008.08115/) (2020)
- [Masked Generative Distillation](https://mlsanity.com/paper/2205.01529/) (2022)
- [Distilling Object Detectors via Decoupled Features](https://mlsanity.com/paper/2103.14475/) (2021)
- [Logit Standardization in Knowledge Distillation](https://mlsanity.com/paper/2403.01427/) (2024)
- [Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity](https://mlsanity.com/paper/2111.14330/) (2021)
- [Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment](https://mlsanity.com/paper/2207.13085/) (2022)
- [D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinement](https://mlsanity.com/paper/2410.13842/) (2024)
- [DEIM: DETR with Improved Matching for Fast Convergence](https://mlsanity.com/paper/2412.04234/) (2024)
- [Class Attention Transfer Based Knowledge Distillation](https://mlsanity.com/paper/2304.12777/) (2023)
- [Localization Distillation for Dense Object Detection](https://mlsanity.com/paper/2102.12252/) (2021)
- [Disentangle Your Dense Object Detector](https://mlsanity.com/paper/2107.02963/) (2021)
- [DETRDistill: A Universal Knowledge Distillation Framework for DETR-families](https://mlsanity.com/paper/2211.10156/) (2022)

## Abstract (arXiv)

Detection Transformers (DETRs) achieve strong performance in object detection but remain challenging to deploy on edge devices due to their high computational cost. Existing DETR distillation methods mainly focus on aligning distillation points, while largely overlooking the quality of the teacher's supervision itself. We observe that due to stage-wise non-monotonic prediction behavior in DETRs, well-localized or correctly classified predictions from earlier stages may degrade in later ones, and some negative predictions become increasingly overconfident. As a result, relying solely on the current stage's predictions yields inaccurate and inconsistent supervision. To address this issue, we propose Teacher Prediction Refinement Distillation (TPRD), a plug-and-play module that refines teacher predictions before distillation by exploiting stage-wise prediction information. TPRD improves supervision quality through Positive Prediction Correction (PPC), which corrects degraded positive predictions by restoring more accurate ones from earlier stages, ensuring reliable localization and classification signals, and Negative Prediction Suppression (NPS) suppresses the influence of overconfident negatives, preventing them from providing misleading supervision to the student. To preserve informative dark knowledge, we further introduce Maximum Dark Knowledge Preservation (MDKP), which selectively refines target-class logits while retaining non-target relations. Extensive experiments on MS COCO and PASCAL VOC demonstrate the effectiveness and robustness of the proposed method. Our code is available at https://github.com/xingyitong1/TPRD.