# Domain-invariant Progressive Knowledge Distillation for UAV-based Object Detection

arXiv: https://arxiv.org/abs/2408.11407 · HTML: https://mlsanity.com/paper/2408.11407/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 21 Aug 2024.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [A Fourier-based Framework for Domain Generalization](https://mlsanity.com/paper/2105.11120/) (2021)
- [DepGraph: Towards Any Structural Pruning](https://mlsanity.com/paper/2301.12900/) (2023)
- [Focal and Global Knowledge Distillation for Detectors](https://mlsanity.com/paper/2111.11837/) (2021)
- [Harmonizing Transferability and Discriminability for Adapting Object Detectors](https://mlsanity.com/paper/2003.06297/) (2020)
- [Self-Training and Adversarial Background Regularization for Unsupervised Domain Adaptive One-Stage Object Detection](https://mlsanity.com/paper/1909.00597/) (2019)
- [Vision Meets Drones: A Challenge](https://mlsanity.com/paper/1804.07437/) (2018)
- [QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection](https://mlsanity.com/paper/2103.09136/) (2021)
- [CrossKD: Cross-Head Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2306.11369/) (2023)
- [PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient](https://mlsanity.com/paper/2207.02039/) (2022)
- [Adaptive Sparse Convolutional Networks with Global Context Enhancement for Faster Object Detection on Drone Images](https://mlsanity.com/paper/2303.14488/) (2023)
- [Deep Learning for UAV-based Object Detection and Tracking: A Survey](https://mlsanity.com/paper/2110.12638/) (2021)
- [SynDrone -- Multi-modal UAV Dataset for Urban Scenarios](https://mlsanity.com/paper/2308.10491/) (2023)
- [Decompose, Adjust, Compose: Effective Normalization by Playing with Frequency for Domain Generalization](https://mlsanity.com/paper/2303.02328/) (2023)
- [TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models](https://mlsanity.com/paper/2301.01296/) (2023)
- [Multi-Source Domain Adaptation for Object Detection](https://mlsanity.com/paper/2106.15793/) (2021)
- [Towards Domain Generalization in Object Detection](https://mlsanity.com/paper/2203.14387/) (2022)
- [Structural Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2211.13133/) (2022)
- [Direct Distillation between Different Domains](https://mlsanity.com/paper/2401.06826/) (2024)
- [Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object Detection](https://mlsanity.com/paper/2308.14286/) (2023)
- [A review of UAV Visual Detection and Tracking Methods](https://mlsanity.com/paper/2306.05089/) (2023)

## Built on by (most cited first, 4 of 4)

- [RemoteSAM: Towards Segment Anything for Earth Observation](https://mlsanity.com/paper/2505.18022/) (2025)
- [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025)
- [Cross-View Open-Vocabulary Object Detection in Aerial Imagery](https://mlsanity.com/paper/2510.03858/) (2025)
- [Disentangle Object and Non-object Infrared Features via Language Guidance](https://mlsanity.com/paper/2601.09228/) (2026)

## Abstract (arXiv)

Knowledge distillation (KD) is an effective method for compressing models in object detection tasks. Due to limited computational capability, UAV-based object detection (UAV-OD) widely adopt the KD technique to obtain lightweight detectors. Existing methods often overlook the significant differences in feature space caused by the large gap in scale between the teacher and student models. This limitation hampers the efficiency of knowledge transfer during the distillation process. Furthermore, the complex backgrounds in UAV images make it challenging for the student model to efficiently learn the object features. In this paper, we propose a novel knowledge distillation framework for UAV-OD. Specifically, a progressive distillation approach is designed to alleviate the feature gap between teacher and student models. Then a new feature alignment method is provided to extract object-related features for enhancing student model's knowledge reception efficiency. Finally, extensive experiments are conducted to validate the effectiveness of our proposed approach. The results demonstrate that our proposed method achieves state-of-the-art (SoTA) performance in two UAV-OD datasets.