mlsanityData to 14 Oct 2026

Data Upcycling Knowledge Distillation for Image Super-Resolution

arXiv 2309.14162 ↗First posted 25 Sep 2023cs.CV cs.AIcited by 4

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Knowledge distillation (KD) compresses deep neural networks by transferring task-related knowledge from cumbersome pre-trained teacher models to compact student models. However, current KD methods for super-resolution (SR) networks overlook the nature of SR task that the outputs of the teacher model are noisy approximations to the ground-truth distribution of high-quality images (GT), which shades the teacher model's knowledge to result in limited KD effects. To utilize the teacher model beyond the GT upper-bound, we present the Data Upcycling Knowledge Distillation (DUKD), to transfer the teacher model's knowledge to the student model through the upcycled in-domain data derived from training data. Besides, we impose label consistency regularization to KD for SR by the paired invertible augmentations to improve the student model's performance and robustness. Comprehensive experiments demonstrate that the DUKD method significantly outperforms previous arts on several SR tasks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 55 of its 85 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

55 of 85 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Supervision Complexity and its Role in Knowledge Distillation28 Jan
  2. 2022Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification18 Aug
  3. 2022Decoupled Knowledge Distillation16 Mar
  4. 2021Does Knowledge Distillation Really Work?10 Jun
  5. 2021Towards Compact Single Image Super-Resolution via Contrastive Self-distillation25 May

Other references · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2019Searching for MobileNetV3cited by 1,994
  8. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,887
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  10. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,627
  11. 2020Knowledge Distillation: A Surveycited by 1,603
  12. 2017Enhanced Deep Residual Networks for Single Image Super-Resolutioncited by 1,552
38 more
  1. 2018Image Super-Resolution Using Very Deep Residual Channel Attention Networkscited by 1,264
  2. 2014Image Super-Resolution Using Deep Convolutional Networkscited by 1,224
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  4. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  5. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,057
  6. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
  7. 2015Accurate Image Super-Resolution Using Very Deep Convolutional Networkscited by 903
  8. 2020Pre-Trained Image Processing Transformercited by 893
  9. 2019Relational Knowledge Distillationcited by 889
  10. 2021Uformer: A General U-Shaped Transformer for Image Restorationcited by 866
  11. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  12. 2021Designing a Practical Degradation Model for Deep Blind Image Super-Resolutioncited by 529
  13. 2019Similarity-Preserving Knowledge Distillationcited by 516
  14. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  15. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 485
  16. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 414
  17. 2019Toward Real-World Single Image Super-Resolution: A New Benchmark and A New Modelcited by 403
  18. 2015Deep Learning with Limited Numerical Precisioncited by 376
  19. 2018Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transformcited by 371
  20. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 347
  21. 2019Structured Knowledge Distillation for Dense Predictioncited by 279
  22. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  23. 2020Component Divide-and-Conquer for Real-World Image Super-Resolutioncited by 246
  24. 2019Residual Non-local Attention Networks for Image Restorationcited by 236
  25. 2019MetaPruning: Meta Learning for Automatic Neural Network Channel Pruningcited by 227
  26. 2019Lightweight Image Super-Resolution with Information Multi-distillation Networkcited by 227
  27. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 215
  28. 2015Structured Pruning of Deep Convolutional Neural Networkscited by 149
  29. 2021Distilling Object Detectors via Decoupled Featurescited by 106
  30. 2020Understanding and Improving Knowledge Distillationcited by 83
  31. 2020Heterogeneous Knowledge Distillation using Information Flow Modelingcited by 73
  32. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 66
  33. 2019Self-Knowledge Distillation in Natural Language Processingcited by 60
  34. 2020Learning with Privileged Information for Efficient Image Super-Resolutioncited by 53
  35. 2021Real-Time Quantized Image Super-Resolution on Mobile NPUs, Mobile AI 2021 Challenge: Reportcited by 34
  36. 2020DAQ: Channel-Wise Distribution-Aware Quantization for Deep Image Super-Resolution Networkscited by 31
  37. 2021Consistency Regularization Can Improve Robustness to Label Noisecited by 18
  38. 2022DCS-RISR: Dynamic Channel Splitting for Efficient Real-world Image Super-Resolutioncited by 2

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 1.

4 papers.

  1. 2024MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolutioncited by 17
  2. 2024U-Know-DiffPAN: An Uncertainty-aware Knowledge Distillation Diffusion Framework with Details Enhancement for PAN-Sharpeningcited by 5
  3. 2024Knowledge Distillation with Multi-granularity Mixture of Priors for Image Super-Resolutioncited by 4
  4. 2025Outlier-Aware Post-Training Quantization for Image Super-Resolutioncited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2022Learning Knowledge Representation with Meta Knowledge Distillation for Single Image Super-Resolution
  2. 2021Local-Selective Feature Distillation for Single Image Super-Resolution
  3. 2023Teacher Network Calibration Improves Cross-Quality Knowledge Distillation
  4. 2024Large-Scale Data-Free Knowledge Distillation for ImageNet via Multi-Resolution Data Generation
  5. 2022Feature-domain Adaptive Contrastive Distillation for Efficient Single Image Super-Resolution
  6. 2024Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration
  7. 2021Follow Your Path: a Progressive Method for Knowledge Distillation
  8. 2024Prior Knowledge Distillation Network for Face Super-Resolution