# ResDiff: Combining CNN and Diffusion Model for Image Super-Resolution

arXiv: https://arxiv.org/abs/2303.08714 · HTML: https://mlsanity.com/paper/2303.08714/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 35 of its 54 references that are in the map.
- First posted on arXiv 15 Mar 2023.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model](https://mlsanity.com/paper/2212.00490/) (2022)
- [Activating More Pixels in Image Super-Resolution Transformer](https://mlsanity.com/paper/2205.04437/) (2022)
- [SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models](https://mlsanity.com/paper/2104.14951/) (2021)
- [Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform](https://mlsanity.com/paper/1804.02815/) (2018)
- [PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models](https://mlsanity.com/paper/2003.03808/) (2020)
- [Deblurring via Stochastic Refinement](https://mlsanity.com/paper/2112.02475/) (2021)
- [Cross Attention Network for Few-shot Classification](https://mlsanity.com/paper/1910.07677/) (2019)
- [SRFlow: Learning the Super-Resolution Space with Normalizing Flow](https://mlsanity.com/paper/2006.14200/) (2020)

## Built on by (most cited first, 30 of 53)

- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [Low-Light Image Enhancement with Wavelet-based Diffusion Models](https://mlsanity.com/paper/2306.00306/) (2023)
- [Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2308.09388/) (2023)
- [Diffusion Models, Image Super-Resolution And Everything: A Survey](https://mlsanity.com/paper/2401.00736/) (2024)
- [DocDiff: Document Enhancement via Residual Diffusion Models](https://mlsanity.com/paper/2305.03892/) (2023)
- [Conditional Image Synthesis with Diffusion Models: A Survey](https://mlsanity.com/paper/2409.19365/) (2024)
- [Wavelet Diffusion Neural Operator](https://mlsanity.com/paper/2412.04833/) (2024)
- [FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models](https://mlsanity.com/paper/2401.15636/) (2024)
- [GCD-DDPM: A Generative Change Detection Model Based on Difference-Feature Guided DDPM](https://mlsanity.com/paper/2306.03424/) (2023)
- [Building Bridges across Spatial and Temporal Resolutions: Reference-Based Super-Resolution via Change Priors and Conditional Diffusion Model](https://mlsanity.com/paper/2403.17460/) (2024)
- [ReFIR: Grounding Large Restoration Models with Retrieval Augmentation](https://mlsanity.com/paper/2410.05601/) (2024)
- [Temporal Residual Guided Diffusion Framework for Event-Driven Video Reconstruction](https://mlsanity.com/paper/2407.10636/) (2024)
- [SAM-DiffSR: Structure-Modulated Diffusion Model for Image Super-Resolution](https://mlsanity.com/paper/2402.17133/) (2024)
- [Diff-Shadow: Global-guided Diffusion Model for Shadow Removal](https://mlsanity.com/paper/2407.16214/) (2024)
- [Reversing Flow for Image Restoration](https://mlsanity.com/paper/2506.16961/) (2025)
- [NAF-DPM: A Nonlinear Activation-Free Diffusion Probabilistic Model for Document Enhancement](https://mlsanity.com/paper/2404.05669/) (2024)
- [LDM3D-VR: Latent Diffusion Model for 3D VR](https://mlsanity.com/paper/2311.03226/) (2023)
- [FLEX: A Backbone for Diffusion-Based Modeling of Spatio-temporal Physical Systems](https://mlsanity.com/paper/2505.17351/) (2025)
- [Solving Diffusion ODEs with Optimal Boundary Conditions for Better Image Super-Resolution](https://mlsanity.com/paper/2305.15357/) (2023)
- [Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models](https://mlsanity.com/paper/2505.20789/) (2025)
- [LSwinSR: UAV Imagery Super-Resolution based on Linear Swin Transformer](https://mlsanity.com/paper/2303.10232/) (2023)
- [One-Step Diffusion-based Real-World Image Super-Resolution with Visual Perception Distillation](https://mlsanity.com/paper/2506.02605/) (2025)
- [Training-Free Model Ensemble for Single-Image Super-Resolution via Strong-Branch Compensation](https://mlsanity.com/paper/2604.11564/) (2026)
- [Reconciling Stochastic and Deterministic Strategies for Zero-shot Image Restoration using Diffusion Model in Dual](https://mlsanity.com/paper/2503.01288/) (2025)
- [The Adversarial AI-Art: Understanding, Generation, Detection, and Benchmarking](https://mlsanity.com/paper/2404.14581/) (2024)
- [High Frequency Matters: Uncertainty Guided Image Compression with Wavelet Diffusion](https://mlsanity.com/paper/2407.12538/) (2024)
- [DiffSR: Learning Radar Reflectivity Synthesis via Diffusion Model from Satellite Observations](https://mlsanity.com/paper/2411.06714/) (2024)
- [MoFRR: Mixture of Diffusion Models for Face Retouching Restoration](https://mlsanity.com/paper/2507.19770/) (2025)
- [Depth-Aware Super-Resolution via Distance-Adaptive Variational Formulation](https://mlsanity.com/paper/2509.05746/) (2025)
- [FiDeSR: High-Fidelity and Detail-Preserving One-Step Diffusion Super-Resolution](https://mlsanity.com/paper/2603.02692/) (2026)

## Abstract (arXiv)

Adapting the Diffusion Probabilistic Model (DPM) for direct image super-resolution is wasteful, given that a simple Convolutional Neural Network (CNN) can recover the main low-frequency content. Therefore, we present ResDiff, a novel Diffusion Probabilistic Model based on Residual structure for Single Image Super-Resolution (SISR). ResDiff utilizes a combination of a CNN, which restores primary low-frequency components, and a DPM, which predicts the residual between the ground-truth image and the CNN predicted image. In contrast to the common diffusion-based methods that directly use LR images to guide the noise towards HR space, ResDiff utilizes the CNN's initial prediction to direct the noise towards the residual space between HR space and CNN-predicted space, which not only accelerates the generation process but also acquires superior sample quality. Additionally, a frequency-domain-based loss function for CNN is introduced to facilitate its restoration, and a frequency-domain guided diffusion is designed for DPM on behalf of predicting high-frequency details. The extensive experiments on multiple benchmark datasets demonstrate that ResDiff outperforms previous diffusion based methods in terms of shorter model convergence time, superior generation quality, and more diverse samples.