# Embedded Self-Distillation in Compact Multi-Branch Ensemble Network for Remote Sensing Scene Classification

arXiv: https://arxiv.org/abs/2104.00222 · HTML: https://mlsanity.com/paper/2104.00222/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 12 of its 58 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification](https://mlsanity.com/paper/1608.05167/) (2016)
- [A Survey on Object Detection in Optical Remote Sensing Images](https://mlsanity.com/paper/1603.06201/) (2016)
- [Multi-Granularity Canonical Appearance Pooling for Remote Sensing Scene Classification](https://mlsanity.com/paper/2004.04491/) (2020)
- [Online Ensemble Model Compression using Knowledge Distillation](https://mlsanity.com/paper/2011.07449/) (2020)
- [Hydra: an Ensemble of Convolutional Neural Networks for Geospatial Land Classification](https://mlsanity.com/paper/1802.03518/) (2018)

## Built on by (most cited first, 8 of 8)

- [An Empirical Study of Remote Sensing Pretraining](https://mlsanity.com/paper/2204.02825/) (2022)
- [LSKNet: A Foundation Lightweight Backbone for Remote Sensing](https://mlsanity.com/paper/2403.11735/) (2024)
- [Consecutive Pretraining: A Knowledge Transfer Learning Strategy with Relevant Unlabeled Data for Remote Sensing Domain](https://mlsanity.com/paper/2207.03860/) (2022)
- [Inherit with Distillation and Evolve with Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory](https://mlsanity.com/paper/2309.15413/) (2023)
- [Location-aware Adaptive Normalization: A Deep Learning Approach For Wildfire Danger Forecasting](https://mlsanity.com/paper/2212.08208/) (2022)
- [Learning at a Glance: Towards Interpretable Data-limited Continual Semantic Segmentation via Semantic-Invariance Modelling](https://mlsanity.com/paper/2407.15429/) (2024)
- [A Robust and Low Complexity Deep Learning Model for Remote Sensing Image Classification](https://mlsanity.com/paper/2211.02820/) (2022)
- [Explainable AI for Earth Observation: Current Methods, Open Challenges, and Opportunities](https://mlsanity.com/paper/2311.04491/) (2023)

## Abstract (arXiv)

Remote sensing (RS) image scene classification task faces many challenges due to the interference from different characteristics of different geographical elements. To solve this problem, we propose a multi-branch ensemble network to enhance the feature representation ability by fusing features in final output logits and intermediate feature maps. However, simply adding branches will increase the complexity of models and decline the inference efficiency. On this issue, we embed self-distillation (SD) method to transfer knowledge from ensemble network to main-branch in it. Through optimizing with SD, main-branch will have close performance as ensemble network. During inference, we can cut other branches to simplify the whole model. In this paper, we first design compact multi-branch ensemble network, which can be trained in an end-to-end manner. Then, we insert SD method on output logits and feature maps. Compared to previous methods, our proposed architecture (ESD-MBENet) performs strongly on classification accuracy with compact design. Extensive experiments are applied on three benchmark RS datasets AID, NWPU-RESISC45 and UC-Merced with three classic baseline models, VGG16, ResNet50 and DenseNet121. Results prove that our proposed ESD-MBENet can achieve better accuracy than previous state-of-the-art (SOTA) complex models. Moreover, abundant visualization analysis make our method more convincing and interpretable.