# Decoder-Free Distillation for Quantized Image Restoration

arXiv: https://arxiv.org/abs/2603.09624 · HTML: https://mlsanity.com/paper/2603.09624/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 40 of its 50 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 40)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [Deep Retinex Decomposition for Low-Light Enhancement](https://mlsanity.com/paper/1808.04560/) (2018)
- [Benchmarking Single Image Dehazing and Beyond](https://mlsanity.com/paper/1712.04143/) (2017)
- [Deep Joint Rain Detection and Removal from a Single Image](https://mlsanity.com/paper/1609.07769/) (2016)
- [Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancement](https://mlsanity.com/paper/2303.06705/) (2023)
- [Getting to Know Low-light Images with The Exclusively Dark Dataset](https://mlsanity.com/paper/1805.11227/) (2018)
- [Learning Enriched Features for Fast Image Restoration and Enhancement](https://mlsanity.com/paper/2205.01649/) (2022)
- [Masked Generative Distillation](https://mlsanity.com/paper/2205.01529/) (2022)
- [LEDNet: Joint Low-light Enhancement and Deblurring in the Dark](https://mlsanity.com/paper/2202.03373/) (2022)
- [PTQ4DiT: Post-training Quantization for Diffusion Transformers](https://mlsanity.com/paper/2405.16005/) (2024)
- [PAMS: Quantized Super-Resolution via Parameterized Max Scale](https://mlsanity.com/paper/2011.04212/) (2020)
- [DarkIR: Robust Low-Light Image Restoration](https://mlsanity.com/paper/2412.13443/) (2024)
- [Learning Conditional Knowledge Distillation for Degraded-Reference Image Quality Assessment](https://mlsanity.com/paper/2108.07948/) (2021)
- [Dynamic Knowledge Distillation for Pre-trained Language Models](https://mlsanity.com/paper/2109.11295/) (2021)
- [DAQ: Channel-Wise Distribution-Aware Quantization for Deep Image Super-Resolution Networks](https://mlsanity.com/paper/2012.11230/) (2020)
- [Extremely Lightweight Quantization Robust Real-Time Single-Image Super Resolution for Mobile Devices](https://mlsanity.com/paper/2105.10288/) (2021)
- [Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformers](https://mlsanity.com/paper/2505.22167/) (2025)
- [Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks](https://mlsanity.com/paper/2009.14502/) (2020)
- [Dynamic Dual Trainable Bounds for Ultra-low Precision Super-Resolution Networks](https://mlsanity.com/paper/2203.03844/) (2022)
- [Fully Quantized Image Super-Resolution Networks](https://mlsanity.com/paper/2011.14265/) (2020)
- [Perceptual Image Enhancement for Smartphone Real-Time Applications](https://mlsanity.com/paper/2210.13552/) (2022)

## Built on by (most cited first, 1 of 1)

- [SPARK: Spatial Policy-driven Adaptive Reinforcement learning for Knowledge distillation](https://mlsanity.com/paper/2606.15243/) (2026)

## Abstract (arXiv)

Quantization-Aware Training (QAT), combined with Knowledge Distillation (KD), holds immense promise for compressing models for edge deployment. However, joint optimization for precision-sensitive image restoration (IR) to recover visual quality from degraded images remains largely underexplored. Directly adapting QAT-KD to low-level vision reveals three critical bottlenecks: teacher-student capacity mismatch, spatial error amplification during decoder distillation, and an optimization "tug-of-war" between reconstruction and distillation losses caused by quantization noise. To tackle these, we introduce Quantization-aware Distilled Restoration (QDR), a framework for edge-deployed IR. QDR eliminates capacity mismatch via FP32 self-distillation and prevents error amplification through Decoder-Free Distillation (DFD), which corrects quantization errors strictly at the network bottleneck. To stabilize the optimization tug-of-war, we propose a Learnable Magnitude Reweighting (LMR) that dynamically balances competing gradients. Finally, we design an Edge-Friendly Model (EFM) featuring a lightweight Learnable Degradation Gating (LDG) to dynamically modulate spatial degradation localization. Extensive experiments across four IR tasks demonstrate that our Int8 model recovers 96.5% of FP32 performance, achieves 442 frames per second (FPS) on an NVIDIA Jetson Orin, and boosts downstream object detection by 16.3 mAP