# LGD: Label-guided Self-distillation for Object Detection

arXiv: https://arxiv.org/abs/2109.11496 · HTML: https://mlsanity.com/paper/2109.11496/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 46 of its 58 references that are in the map.
- First posted on arXiv 23 Sep 2021.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection](https://mlsanity.com/paper/1912.02424/) (2019)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [CrowdHuman: A Benchmark for Detecting Human in a Crowd](https://mlsanity.com/paper/1805.00123/) (2018)
- [Distilling Object Detectors with Fine-grained Feature Imitation](https://mlsanity.com/paper/1906.03609/) (2019)
- [Knowledge Distillation by On-the-Fly Native Ensemble](https://mlsanity.com/paper/1806.04606/) (2018)

## Built on by (most cited first, 19 of 19)

- [Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2303.05892/) (2023)
- [ConQueR: Query Contrast Voxel-DETR for 3D Object Detection](https://mlsanity.com/paper/2212.07289/) (2022)
- [D3ETR: Decoder Distillation for Detection Transformer](https://mlsanity.com/paper/2211.09768/) (2022)
- [Object-Centric Instruction Augmentation for Robotic Manipulation](https://mlsanity.com/paper/2401.02814/) (2024)
- [Panoramic Video Salient Object Detection with Ambisonic Audio Guidance](https://mlsanity.com/paper/2211.14419/) (2022)
- [LabelDistill: Label-guided Cross-modal Knowledge Distillation for Camera-based 3D Object Detection](https://mlsanity.com/paper/2407.10164/) (2024)
- [Label-Guided Auxiliary Training Improves 3D Object Detector](https://mlsanity.com/paper/2207.11753/) (2022)
- [Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object Detection](https://mlsanity.com/paper/2308.14286/) (2023)
- [Knowledge Amalgamation for Object Detection with Transformers](https://mlsanity.com/paper/2203.03187/) (2022)
- [Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentation](https://mlsanity.com/paper/2407.13254/) (2024)
- [RdimKD: Generic Distillation Paradigm by Dimensionality Reduction](https://mlsanity.com/paper/2312.08700/) (2023)
- [Smooth and Stepwise Self-Distillation for Object Detection](https://mlsanity.com/paper/2303.05015/) (2023)
- [Volumetric medical image segmentation through dual self-distillation in U-shaped networks](https://mlsanity.com/paper/2306.03271/) (2023)
- [SEFormer: Structure Embedding Transformer for 3D Object Detection](https://mlsanity.com/paper/2209.01745/) (2022)
- [Spatial-wise Dynamic Distillation for MLP-like Efficient Visual Fault Detection of Freight Trains](https://mlsanity.com/paper/2312.05832/) (2023)
- [Unveiling the Black Box: Independent Functional Module Evaluation for Bird's-Eye-View Perception Model](https://mlsanity.com/paper/2409.11969/) (2024)
- [Designing Object Detection Models for TinyML: Foundations, Comparative Analysis, Challenges, and Emerging Solutions](https://mlsanity.com/paper/2508.08352/) (2025)
- [Efficient Visual Fault Detection for Freight Train Braking System via Heterogeneous Self Distillation in the Wild](https://mlsanity.com/paper/2307.00701/) (2023)
- [Real-Time 3D Object Detection with Inference-Aligned Learning](https://mlsanity.com/paper/2511.16140/) (2025)

## Abstract (arXiv)

In this paper, we propose the first self-distillation framework for general object detection, termed LGD (Label-Guided self-Distillation). Previous studies rely on a strong pretrained teacher to provide instructive knowledge that could be unavailable in real-world scenarios. Instead, we generate an instructive knowledge based only on student representations and regular labels. Our framework includes sparse label-appearance encoder, inter-object relation adapter and intra-object knowledge mapper that jointly form an implicit teacher at training phase, dynamically dependent on labels and evolving student representations. They are trained end-to-end with detector and discarded in inference. Experimentally, LGD obtains decent results on various detectors, datasets, and extensive tasks like instance segmentation. For example in MS-COCO dataset, LGD improves RetinaNet with ResNet-50 under 2x single-scale training from 36.2% to 39.0% mAP (+ 2.8%). It boosts much stronger detectors like FCOS with ResNeXt-101 DCN v2 under 2x multi-scale training from 46.1% to 47.9% (+ 1.8%). Compared with a classical teacher-based method FGFI, LGD not only performs better without requiring pretrained teacher but also reduces 51% training cost beyond inherent student learning. Codes are available at https://github.com/megvii-research/LGD.