mlsanityData to 7 Oct 2026

OD-DETR: Online Distillation for Stabilizing Training of Detection Transformer

arXiv 2406.05791 ↗First posted 9 Jun 2024cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

DEtection TRansformer (DETR) becomes a dominant paradigm, mainly due to its common architecture with high accuracy and no post-processing. However, DETR suffers from unstable training dynamics. It consumes more data and epochs to converge compared with CNN-based detectors. This paper aims to stabilize DETR training through the online distillation. It utilizes a teacher model, accumulated by Exponential Moving Average (EMA), and distills its knowledge into the online model in following three aspects. First, the matching relation between object queries and ground truth (GT) boxes in the teacher is employed to guide the student, so queries within the student are not only assigned labels based on their own predictions, but also refer to the matching results from the teacher. Second, the teacher's initial query is given to the online student, and its prediction is directly constrained by the corresponding output from the teacher. Finally, the object queries from teacher's different decoding stages are used to build the auxiliary groups to accelerate the convergence. For each GT, two queries with the least matching costs are selected into this extra group, and they predict the GT box and participate the optimization. Extensive experiments show that the proposed OD-DETR successfully stabilizes the training, and significantly increases the performance without bringing in more parameters.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 34 of its 42 references that are in the map. First posted on arXiv 9 Jun 2024.

Builds on

34 of 42 references.

References · 34

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  8. 2015Fast R-CNNcited by 2,930
  9. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  10. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  11. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  12. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
22 more
  1. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  2. 2019CenterNet: Keypoint Triplets for Object Detectioncited by 654
  3. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 569
  4. 2020Sparse R-CNN: End-to-End Object Detection with Learnable Proposalscited by 494
  5. 2022DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingcited by 486
  6. 2021Conditional DETR for Fast Training Convergencecited by 422
  7. 2020Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detectioncited by 408
  8. 2021End-to-End Semi-Supervised Object Detection with Soft Teachercited by 315
  9. 2021Unbiased Teacher for Semi-Supervised Object Detectioncited by 305
  10. 2019RepPoints: Point Set Representation for Object Detectioncited by 264
  11. 2021TOOD: Task-aligned One-stage Object Detectioncited by 218
  12. 2021Fast Convergence of DETR with Spatially Modulated Co-Attentioncited by 176
  13. 2022DETRs with Hybrid Matchingcited by 149
  14. 2021Efficient DETR: Improving End-to-End Object Detector with Dense Priorcited by 128
  15. 2021Humble Teachers Teach Better Students for Semi-Supervised Object Detectioncited by 115
  16. 2020OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learningcited by 65
  17. 2023Detection Transformer with Stable Matchingcited by 43
  18. 2022DETRDistill: A Universal Knowledge Distillation Framework for DETR-familiescited by 36
  19. 2022D3ETR: Decoder Distillation for Detection Transformercited by 17
  20. 2023StageInteractor: Query-based Object Detector with Cross-stage Interactioncited by 8
  21. 2022IoU-Enhanced Attention for End-to-End Task Specific Object Detectioncited by 2
  22. 2023RecursiveDet: End-to-End Region-based Recursive Object Detectioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Architectural Insights into Knowledge Distillation for Object Detection: A Comprehensive Reviewcited by 1
  2. 2026iFAN: Inference-Aware Learning for Plain Mask Transformerscited by 0
  3. 2026Enhanced Knowledge Distillation for Detection Transformer via Teacher Prediction Refinementcited by 0
  4. 2026Match One, Learn with Graph: One-to-Graph Query Collaboration with Backward Sharing for Object Detectioncited by 0