# Towards Compact Single Image Super-Resolution via Contrastive Self-distillation

arXiv: https://arxiv.org/abs/2105.11683 · HTML: https://mlsanity.com/paper/2105.11683/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 30 of its 38 references that are in the map.
- First posted on arXiv 25 May 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 30)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [Deeply-Recursive Convolutional Network for Image Super-Resolution](https://mlsanity.com/paper/1511.04491/) (2015)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network](https://mlsanity.com/paper/1803.08664/) (2018)
- [Lightweight Image Super-Resolution with Information Multi-distillation Network](https://mlsanity.com/paper/1909.11856/) (2019)
- [Universally Slimmable Networks and Improved Training Techniques](https://mlsanity.com/paper/1903.05134/) (2019)
- [Contrastive Learning for Compact Single Image Dehazing](https://mlsanity.com/paper/2104.09367/) (2021)
- [Feedback Network for Image Super-Resolution](https://mlsanity.com/paper/1903.09814/) (2019)
- [Knowledge Distillation Meets Self-Supervision](https://mlsanity.com/paper/2006.07114/) (2020)
- [Efficient Image Super-Resolution Using Pixel Attention](https://mlsanity.com/paper/2010.01073/) (2020)
- [PAMS: Quantized Super-Resolution via Parameterized Max Scale](https://mlsanity.com/paper/2011.04212/) (2020)
- [AdderSR: Towards Energy Efficient Image Super-Resolution](https://mlsanity.com/paper/2009.08891/) (2020)
- [Efficient Super Resolution Using Binarized Neural Network](https://mlsanity.com/paper/1812.06378/) (2018)
- [DRFN: Deep Recurrent Fusion Network for Single-Image Super-Resolution with Large Factors](https://mlsanity.com/paper/1908.08837/) (2019)
- [MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillation](https://mlsanity.com/paper/2008.12094/) (2020)
- [GhostSR: Learning Ghost Features for Efficient Image Super-Resolution](https://mlsanity.com/paper/2101.08525/) (2021)
- [Anytime Inference with Distilled Hierarchical Neural Ensembles](https://mlsanity.com/paper/2003.01474/) (2020)

## Built on by (most cited first, 24 of 24)

- [Contrastive Semi-supervised Learning for Underwater Image Restoration via Reliable Bank](https://mlsanity.com/paper/2303.09101/) (2023)
- [RainMamba: Enhanced Locality Learning with State Space Models for Video Deraining](https://mlsanity.com/paper/2407.21773/) (2024)
- [Residual Local Feature Network for Efficient Super-Resolution](https://mlsanity.com/paper/2205.07514/) (2022)
- [A Practical Contrastive Learning Framework for Single-Image Super-Resolution](https://mlsanity.com/paper/2111.13924/) (2021)
- [SCS-Co: Self-Consistent Style Contrastive Learning for Image Harmonization](https://mlsanity.com/paper/2204.13962/) (2022)
- [ARM: Any-Time Super-Resolution Method](https://mlsanity.com/paper/2203.10812/) (2022)
- [Efficient and Accurate Quantized Image Super-Resolution on Mobile NPUs, Mobile AI & AIM 2022 challenge: Report](https://mlsanity.com/paper/2211.05910/) (2022)
- [Semi-Supervised Video Desnowing Network via Temporal Decoupling Experts and Distribution-Driven Contrastive Regularization](https://mlsanity.com/paper/2410.07901/) (2024)
- [SAPNet: Segmentation-Aware Progressive Network for Perceptual Contrastive Deraining](https://mlsanity.com/paper/2111.08892/) (2021)
- [Learning from Multi-Perception Features for Real-Word Image Super-resolution](https://mlsanity.com/paper/2305.18547/) (2023)
- [Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration](https://mlsanity.com/paper/2412.08939/) (2024)
- [Data Upcycling Knowledge Distillation for Image Super-Resolution](https://mlsanity.com/paper/2309.14162/) (2023)
- [Knowledge Distillation with Multi-granularity Mixture of Priors for Image Super-Resolution](https://mlsanity.com/paper/2404.02573/) (2024)
- [Residual Contrastive Learning for Image Reconstruction: Learning Transferable Representations from Noisy Images](https://mlsanity.com/paper/2106.10070/) (2021)
- [Noise-free Optimization in Early Training Steps for Image Super-Resolution](https://mlsanity.com/paper/2312.17526/) (2023)
- [Contrastive Learning for Local and Global Learning MRI Reconstruction](https://mlsanity.com/paper/2111.15200/) (2021)
- [DCS-RISR: Dynamic Channel Splitting for Efficient Real-world Image Super-Resolution](https://mlsanity.com/paper/2212.07613/) (2022)
- [Suppressing Uncertainties in Degradation Estimation for Blind Super-Resolution](https://mlsanity.com/paper/2406.16459/) (2024)
- [Learning Knowledge Representation with Meta Knowledge Distillation for Single Image Super-Resolution](https://mlsanity.com/paper/2207.08356/) (2022)
- [Feature-domain Adaptive Contrastive Distillation for Efficient Single Image Super-Resolution](https://mlsanity.com/paper/2211.15951/) (2022)
- [IRConStyle: Image Restoration Framework Using Contrastive Learning and Style Transfer](https://mlsanity.com/paper/2402.15784/) (2024)
- [A Hybrid CNN-Transformer Architecture with Frequency Domain Contrastive Learning for Image Deraining](https://mlsanity.com/paper/2308.03340/) (2023)
- [Unpaired MRI Super Resolution with Contrastive Learning](https://mlsanity.com/paper/2310.15767/) (2023)
- [Flow-Map Distillation on Relation Manifolds for Image Restoration](https://mlsanity.com/paper/2608.05769/) (2026)

## Abstract (arXiv)

Convolutional neural networks (CNNs) are highly successful for super-resolution (SR) but often require sophisticated architectures with heavy memory cost and computational overhead, significantly restricts their practical deployments on resource-limited devices. In this paper, we proposed a novel contrastive self-distillation (CSD) framework to simultaneously compress and accelerate various off-the-shelf SR models. In particular, a channel-splitting super-resolution network can first be constructed from a target teacher network as a compact student network. Then, we propose a novel contrastive loss to improve the quality of SR images and PSNR/SSIM via explicit knowledge transfer. Extensive experiments demonstrate that the proposed CSD scheme effectively compresses and accelerates several standard SR models such as EDSR, RCAN and CARN. Code is available at https://github.com/Booooooooooo/CSD.