# Local-Selective Feature Distillation for Single Image Super-Resolution

arXiv: https://arxiv.org/abs/2111.10988 · HTML: https://mlsanity.com/paper/2111.10988/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 22 Nov 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Accelerating the Super-Resolution Convolutional Neural Network](https://mlsanity.com/paper/1608.00367/) (2016)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [Deeply-Recursive Convolutional Network for Image Super-Resolution](https://mlsanity.com/paper/1511.04491/) (2015)
- [Single Image Super-Resolution via a Holistic Attention Network](https://mlsanity.com/paper/2008.08767/) (2020)
- [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018)
- [MemNet: A Persistent Memory Network for Image Restoration](https://mlsanity.com/paper/1708.02209/) (2017)
- [Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons](https://mlsanity.com/paper/1811.03233/) (2018)
- [Residual Non-local Attention Networks for Image Restoration](https://mlsanity.com/paper/1903.10082/) (2019)
- [Deep Back-Projection Networks For Super-Resolution](https://mlsanity.com/paper/1803.02735/) (2018)
- [Distilling Object Detectors with Fine-grained Feature Imitation](https://mlsanity.com/paper/1906.03609/) (2019)
- [Non-Local Recurrent Network for Image Restoration](https://mlsanity.com/paper/1806.02919/) (2018)
- [Deep Learning for Single Image Super-Resolution: A Brief Review](https://mlsanity.com/paper/1808.03344/) (2018)
- [Interpreting Super-Resolution Networks with Local Attribution Maps](https://mlsanity.com/paper/2011.11036/) (2020)
- [General Instance Distillation for Object Detection](https://mlsanity.com/paper/2103.02340/) (2021)
- [Fourier Space Losses for Efficient Perceptual Image Super-Resolution](https://mlsanity.com/paper/2106.00783/) (2021)
- [Feature-map-level Online Adversarial Knowledge Distillation](https://mlsanity.com/paper/2002.01775/) (2020)
- [Learning with Privileged Information for Efficient Image Super-Resolution](https://mlsanity.com/paper/2007.07524/) (2020)

## Built on by (most cited first, 4 of 4)

- [Learning to Distill Global Representation for Sparse-View CT](https://mlsanity.com/paper/2308.08463/) (2023)
- [PyNET-QxQ: An Efficient PyNET Variant for QxQ Bayer Pattern Demosaicing in CMOS Image Sensors](https://mlsanity.com/paper/2203.04314/) (2022)
- [Pairwise Distance Distillation for Unsupervised Real-World Image Super-Resolution](https://mlsanity.com/paper/2407.07302/) (2024)
- [Feature-domain Adaptive Contrastive Distillation for Efficient Single Image Super-Resolution](https://mlsanity.com/paper/2211.15951/) (2022)

## Abstract (arXiv)

Recent improvements in convolutional neural network (CNN)-based single image super-resolution (SISR) methods rely heavily on fabricating network architectures, rather than finding a suitable training algorithm other than simply minimizing the regression loss. Adapting knowledge distillation (KD) can open a way for bringing further improvement for SISR, and it is also beneficial in terms of model efficiency. KD is a model compression method that improves the performance of Deep Neural Networks (DNNs) without using additional parameters for testing. It is getting the limelight recently for its competence at providing a better capacity-performance tradeoff. In this paper, we propose a novel feature distillation (FD) method which is suitable for SISR. We show the limitations of the existing FitNet-based FD method that it suffers in the SISR task, and propose to modify the existing FD algorithm to focus on local feature information. In addition, we propose a teacher-student-difference-based soft feature attention method that selectively focuses on specific pixel locations to extract feature information. We call our method local-selective feature distillation (LSFD) and verify that our method outperforms conventional FD methods in SISR problems.