# Applications of Knowledge Distillation in Remote Sensing: A Survey

arXiv: https://arxiv.org/abs/2409.12111 · HTML: https://mlsanity.com/paper/2409.12111/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 106 of its 386 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 30 of 106)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [Distilling Knowledge via Knowledge Review](https://mlsanity.com/paper/2104.09044/) (2021)
- [iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images](https://mlsanity.com/paper/1905.12886/) (2019)
- [The Computational Limits of Deep Learning](https://mlsanity.com/paper/2007.05558/) (2020)
- [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021)
- [Ensemble Distribution Distillation](https://mlsanity.com/paper/1905.00076/) (2019)
- [Large scale distributed neural network training through online distillation](https://mlsanity.com/paper/1804.03235/) (2018)
- [Cross-Layer Distillation with Semantic Calibration](https://mlsanity.com/paper/2012.03236/) (2020)
- [Knowledge Distillation Meets Self-Supervision](https://mlsanity.com/paper/2006.07114/) (2020)
- [Dataset Distillation: A Comprehensive Review](https://mlsanity.com/paper/2301.07014/) (2023)
- [Soft-Label Dataset Distillation and Text Dataset Distillation](https://mlsanity.com/paper/1910.02551/) (2019)
- [Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy](https://mlsanity.com/paper/1711.05852/) (2017)
- [Data Distillation: A Survey](https://mlsanity.com/paper/2301.04272/) (2023)
- [Heterogeneous Knowledge Distillation using Information Flow Modeling](https://mlsanity.com/paper/2005.00727/) (2020)
- [From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels](https://mlsanity.com/paper/2303.13005/) (2023)
- [The Multi-Temporal Urban Development SpaceNet Dataset](https://mlsanity.com/paper/2102.04420/) (2021)
- [Agriculture-Vision: A Large Aerial Image Database for Agricultural Pattern Analysis](https://mlsanity.com/paper/2001.01306/) (2020)
- [Online Knowledge Distillation for Efficient Pose Estimation](https://mlsanity.com/paper/2108.02092/) (2021)
- [Multi-Modal Temporal Attention Models for Crop Mapping from Satellite Time Series](https://mlsanity.com/paper/2112.07558/) (2021)
- [RarePlanes: Synthetic Data Takes Flight](https://mlsanity.com/paper/2006.02963/) (2020)
- [SpaceNet 6: Multi-Sensor All Weather Mapping Dataset](https://mlsanity.com/paper/2004.06500/) (2020)
- [A Comprehensive Survey on Knowledge Distillation of Diffusion Models](https://mlsanity.com/paper/2304.04262/) (2023)
- [Knowledge Distillation for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/1911.03462/) (2019)
- [Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledge](https://mlsanity.com/paper/2004.00176/) (2020)
- [Knowledge Transfer via Dense Cross-Layer Mutual-Distillation](https://mlsanity.com/paper/2008.07816/) (2020)
- [Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data](https://mlsanity.com/paper/2108.05698/) (2021)
- [PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptation](https://mlsanity.com/paper/2208.10160/) (2022)
- [Understanding the Role of the Projector in Knowledge Distillation](https://mlsanity.com/paper/2303.11098/) (2023)

## Built on by (most cited first, 8 of 8)

- [MSSDF: Modality-Shared Self-supervised Distillation for High-Resolution Multi-modal Remote Sensing Image Learning](https://mlsanity.com/paper/2506.09327/) (2025)
- [Lightweight Cloud Masking Models for On-Board Inference in Hyperspectral Imaging](https://mlsanity.com/paper/2507.08052/) (2025)
- [Adaptive Dual-Mode Distillation with Incentive Schemes for Scalable, Heterogeneous Federated Learning on Non-IID Data](https://mlsanity.com/paper/2509.22507/) (2025)
- [TinyNeRV: Compact Neural Video Representations via Capacity Scaling, Distillation, and Low-Precision Inference](https://mlsanity.com/paper/2604.09220/) (2026)
- [Cross-Domain Dead Tree Detection via Knowledge Distillation in Aerial Imagery](https://mlsanity.com/paper/2606.02303/) (2026)
- [Dual-Selective Network for Domain-Incremental Change Detection](https://mlsanity.com/paper/2607.02299/) (2026)
- [Distilling deep optical flow stereo methods to retrieve dense three-dimensional wind fields](https://mlsanity.com/paper/2609.03100/) (2026)
- [Cross-Architecture Foundation-Model Distillation for Edge Flood Segmentation](https://mlsanity.com/paper/2609.20441/) (2026)

## Abstract (arXiv)

With the ever-growing complexity of models in the field of remote sensing (RS), there is an increasing demand for solutions that balance model accuracy with computational efficiency. Knowledge distillation (KD) has emerged as a powerful tool to meet this need, enabling the transfer of knowledge from large, complex models to smaller, more efficient ones without significant loss in performance. This review article provides an extensive examination of KD and its innovative applications in RS. KD, a technique developed to transfer knowledge from a complex, often cumbersome model (teacher) to a more compact and efficient model (student), has seen significant evolution and application across various domains. Initially, we introduce the fundamental concepts and historical progression of KD methods. The advantages of employing KD are highlighted, particularly in terms of model compression, enhanced computational efficiency, and improved performance, which are pivotal for practical deployments in RS scenarios. The article provides a comprehensive taxonomy of KD techniques, where each category is critically analyzed to demonstrate the breadth and depth of the alternative options, and illustrates specific case studies that showcase the practical implementation of KD methods in RS tasks, such as instance segmentation and object detection. Further, the review discusses the challenges and limitations of KD in RS, including practical constraints and prospective future directions, providing a comprehensive overview for researchers and practitioners in the field of RS. Through this organization, the paper not only elucidates the current state of research in KD but also sets the stage for future research opportunities, thereby contributing significantly to both academic research and real-world applications.