mlsanityData to 14 Oct 2026

Towards Compact Single Image Super-Resolution via Contrastive Self-distillation

arXiv 2105.11683 ↗First posted 25 May 2021cs.CVcited by 24

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Convolutional neural networks (CNNs) are highly successful for super-resolution (SR) but often require sophisticated architectures with heavy memory cost and computational overhead, significantly restricts their practical deployments on resource-limited devices. In this paper, we proposed a novel contrastive self-distillation (CSD) framework to simultaneously compress and accelerate various off-the-shelf SR models. In particular, a channel-splitting super-resolution network can first be constructed from a target teacher network as a compact student network. Then, we propose a novel contrastive loss to improve the quality of SR images and PSNR/SSIM via explicit knowledge transfer. Extensive experiments demonstrate that the proposed CSD scheme effectively compresses and accelerates several standard SR models such as EDSR, RCAN and CARN. Code is available at https://github.com/Booooooooooo/CSD.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 30 of its 38 references that are in the map. First posted on arXiv 25 May 2021.

Builds on

30 of 38 references.

References · 30

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  7. 2017Enhanced Deep Residual Networks for Single Image Super-Resolutioncited by 1,552
  8. 2018Image Super-Resolution Using Very Deep Residual Channel Attention Networkscited by 1,264
  9. 2018ESRGAN: Enhanced Super-Resolution Generative Adversarial Networkscited by 1,254
  10. 2019Contrastive Representation Distillationcited by 748
  11. 2020Contrastive Learning for Unpaired Image-to-Image Translationcited by 639
  12. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
18 more
  1. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  2. 2019On the Efficacy of Knowledge Distillationcited by 381
  3. 2015Deeply-Recursive Convolutional Network for Image Super-Resolutioncited by 317
  4. 2018Slimmable Neural Networkscited by 305
  5. 2018Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Networkcited by 264
  6. 2019Lightweight Image Super-Resolution with Information Multi-distillation Networkcited by 227
  7. 2019Universally Slimmable Networks and Improved Training Techniquescited by 221
  8. 2021Contrastive Learning for Compact Single Image Dehazingcited by 219
  9. 2019Feedback Network for Image Super-Resolutioncited by 155
  10. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  11. 2020Efficient Image Super-Resolution Using Pixel Attentioncited by 87
  12. 2020PAMS: Quantized Super-Resolution via Parameterized Max Scalecited by 48
  13. 2020AdderSR: Towards Energy Efficient Image Super-Resolutioncited by 32
  14. 2018Efficient Super Resolution Using Binarized Neural Networkcited by 29
  15. 2019DRFN: Deep Recurrent Fusion Network for Single-Image Super-Resolution with Large Factorscited by 24
  16. 2020MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillationcited by 18
  17. 2021GhostSR: Learning Ghost Features for Efficient Image Super-Resolutioncited by 8
  18. 2020Anytime Inference with Distilled Hierarchical Neural Ensemblescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 7; 2023: 6; 2024: 6; 2025: 0; 2026: 1.

24 papers.

Sort:

By area:

  1. 2023Contrastive Semi-supervised Learning for Underwater Image Restoration via Reliable Bankcited by 71
  2. 2024RainMamba: Enhanced Locality Learning with State Space Models for Video Derainingcited by 56
  3. 2022Residual Local Feature Network for Efficient Super-Resolutioncited by 53
  4. 2021A Practical Contrastive Learning Framework for Single-Image Super-Resolutioncited by 30
  5. 2022SCS-Co: Self-Consistent Style Contrastive Learning for Image Harmonizationcited by 24
  6. 2022ARM: Any-Time Super-Resolution Methodcited by 19
  7. 2022Efficient and Accurate Quantized Image Super-Resolution on Mobile NPUs, Mobile AI & AIM 2022 challenge: Reportcited by 16
  8. 2024Semi-Supervised Video Desnowing Network via Temporal Decoupling Experts and Distribution-Driven Contrastive Regularizationcited by 11
  9. 2021SAPNet: Segmentation-Aware Progressive Network for Perceptual Contrastive Derainingcited by 9
  10. 2023Learning from Multi-Perception Features for Real-Word Image Super-resolutioncited by 9
  11. 2024Dynamic Contrastive Knowledge Distillation for Efficient Image Restorationcited by 6
  12. 2023Data Upcycling Knowledge Distillation for Image Super-Resolutioncited by 4
12 more
  1. 2024Knowledge Distillation with Multi-granularity Mixture of Priors for Image Super-Resolutioncited by 4
  2. 2021Residual Contrastive Learning for Image Reconstruction: Learning Transferable Representations from Noisy Imagescited by 3
  3. 2023Noise-free Optimization in Early Training Steps for Image Super-Resolutioncited by 3
  4. 2021Contrastive Learning for Local and Global Learning MRI Reconstructioncited by 2
  5. 2022DCS-RISR: Dynamic Channel Splitting for Efficient Real-world Image Super-Resolutioncited by 2
  6. 2024Suppressing Uncertainties in Degradation Estimation for Blind Super-Resolutioncited by 2
  7. 2022Learning Knowledge Representation with Meta Knowledge Distillation for Single Image Super-Resolutioncited by 1
  8. 2022Feature-domain Adaptive Contrastive Distillation for Efficient Single Image Super-Resolutioncited by 1
  9. 2024IRConStyle: Image Restoration Framework Using Contrastive Learning and Style Transfercited by 1
  10. 2023A Hybrid CNN-Transformer Architecture with Frequency Domain Contrastive Learning for Image Derainingcited by 0
  11. 2023Unpaired MRI Super Resolution with Contrastive Learningcited by 0
  12. 2026Flow-Map Distillation on Relation Manifolds for Image Restorationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Compressing Deep Image Super-resolution Models
  2. 2024One Step Diffusion-based Super-Resolution with Time-Aware Distillation
  3. 2021Local-Selective Feature Distillation for Single Image Super-Resolution
  4. 2026PhoenixSR: Generative Heterogeneous Distillation Unleashes Efficient Models for Real-World Super-Resolution
  5. 2023DDistill-SR: Reparameterized Dynamic Distillation Network for Lightweight Image Super-Resolution
  6. 2021Self-Supervised Adaptation for Video Super-Resolution
  7. 2024HASN: Hybrid Attention Separable Network for Efficient Image Super-resolution
  8. 2022Hitchhiker's Guide to Super-Resolution: Introduction and Recent Advances