# Feature-domain Adaptive Contrastive Distillation for Efficient Single Image Super-Resolution

arXiv: https://arxiv.org/abs/2211.15951 · HTML: https://mlsanity.com/paper/2211.15951/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 30 of its 64 references that are in the map.
- First posted on arXiv 29 Nov 2022.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [Deeply-Recursive Convolutional Network for Image Super-Resolution](https://mlsanity.com/paper/1511.04491/) (2015)
- [DetCo: Unsupervised Contrastive Learning for Object Detection](https://mlsanity.com/paper/2102.04803/) (2021)
- [GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution](https://mlsanity.com/paper/2012.00739/) (2020)
- [Adversarial Self-Supervised Contrastive Learning](https://mlsanity.com/paper/2006.07589/) (2020)
- [Learning with Privileged Information for Efficient Image Super-Resolution](https://mlsanity.com/paper/2007.07524/) (2020)
- [Complementary Relation Contrastive Distillation](https://mlsanity.com/paper/2103.16367/) (2021)
- [Distilling Knowledge From a Deep Pose Regressor Network](https://mlsanity.com/paper/1908.00858/) (2019)
- [Blind Image Super-Resolution via Contrastive Representation Learning](https://mlsanity.com/paper/2107.00708/) (2021)
- [A Practical Contrastive Learning Framework for Single-Image Super-Resolution](https://mlsanity.com/paper/2111.13924/) (2021)
- [Towards Compact Single Image Super-Resolution via Contrastive Self-distillation](https://mlsanity.com/paper/2105.11683/) (2021)
- [PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation](https://mlsanity.com/paper/2106.14681/) (2021)
- [GLEAN: Generative Latent Bank for Image Super-Resolution and Beyond](https://mlsanity.com/paper/2207.14812/) (2022)
- [Fast Nearest Convolution for Real-Time Efficient Image Super-Resolution](https://mlsanity.com/paper/2208.11609/) (2022)
- [Local-Selective Feature Distillation for Single Image Super-Resolution](https://mlsanity.com/paper/2111.10988/) (2021)

## Built on by (most cited first, 1 of 1)

- [Real-Time 4K Super-Resolution of Compressed AVIF Images. AIS 2024 Challenge Survey](https://mlsanity.com/paper/2404.16484/) (2024)

## Abstract (arXiv)

Recently, CNN-based SISR has numerous parameters and high computational cost to achieve better performance, limiting its applicability to resource-constrained devices such as mobile. As one of the methods to make the network efficient, Knowledge Distillation (KD), which transfers teacher's useful knowledge to student, is currently being studied. More recently, KD for SISR utilizes Feature Distillation (FD) to minimize the Euclidean distance loss of feature maps between teacher and student networks, but it does not sufficiently consider how to effectively and meaningfully deliver knowledge from teacher to improve the student performance at given network capacity constraints. In this paper, we propose a feature-domain adaptive contrastive distillation (FACD) method for efficiently training lightweight student SISR networks. We show the limitations of the existing FD methods using Euclidean distance loss, and propose a feature-domain contrastive loss that makes a student network learn richer information from the teacher's representation in the feature domain. In addition, we propose an adaptive distillation that selectively applies distillation depending on the conditions of the training patches. The experimental results show that the student EDSR and RCAN networks with the proposed FACD scheme improves not only the PSNR performance of the entire benchmark datasets and scales, but also the subjective image quality compared to the conventional FD approaches.