# DiffIR: Efficient Diffusion Model for Image Restoration

arXiv: https://arxiv.org/abs/2303.09472 · HTML: https://mlsanity.com/paper/2303.09472/

## Key facts

- Cited by 255 papers in the map (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 74 of its 89 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 30 of 74)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)

## Built on by (most cited first, 30 of 255)

- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [In-Context LoRA for Diffusion Transformers](https://mlsanity.com/paper/2410.23775/) (2024)
- [Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution](https://mlsanity.com/paper/2312.06640/) (2023)
- [Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2308.09388/) (2023)
- [3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priors](https://mlsanity.com/paper/2410.16266/) (2024)
- [Principled Probabilistic Imaging using Diffusion Models as Plug-and-Play Priors](https://mlsanity.com/paper/2405.18782/) (2024)
- [VmambaIR: Visual State Space Model for Image Restoration](https://mlsanity.com/paper/2403.11423/) (2024)
- [Hierarchical Integration Diffusion Model for Realistic Image Deblurring](https://mlsanity.com/paper/2305.12966/) (2023)
- [Arbitrary-steps Image Super-resolution via Diffusion Inversion](https://mlsanity.com/paper/2412.09013/) (2024)
- [Diffusion Models in Low-Level Vision: A Survey](https://mlsanity.com/paper/2406.11138/) (2024)
- [Reti-Diff: Illumination Degradation Image Restoration with Retinex-based Latent Diffusion Model](https://mlsanity.com/paper/2311.11638/) (2023)
- [Degradation-Guided One-Step Image Super-Resolution with Diffusion Priors](https://mlsanity.com/paper/2409.17058/) (2024)
- [AddSR: Accelerating Diffusion-based Blind Super-Resolution with Adversarial Diffusion Distillation](https://mlsanity.com/paper/2404.01717/) (2024)
- [Diffusion Models, Image Super-Resolution And Everything: A Survey](https://mlsanity.com/paper/2401.00736/) (2024)
- [Efficient Diffusion Model for Image Restoration by Residual Shifting](https://mlsanity.com/paper/2403.07319/) (2024)
- [Perceive-IR: Learning to Perceive Degradation Better for All-in-One Image Restoration](https://mlsanity.com/paper/2408.15994/) (2024)
- [EDiffSR: An Efficient Diffusion Probabilistic Model for Remote Sensing Image Super-Resolution](https://mlsanity.com/paper/2310.19288/) (2023)
- [Boosting Diffusion Models with Moving Average Sampling in Frequency Domain](https://mlsanity.com/paper/2403.17870/) (2024)
- [Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks](https://mlsanity.com/paper/2403.00644/) (2024)
- [UniRestore: Unified Perceptual and Task-Oriented Image Restoration Model Using Diffusion Prior](https://mlsanity.com/paper/2501.13134/) (2025)
- [FoundIR: Unleashing Million-scale Training Data to Advance Foundation Models for Image Restoration](https://mlsanity.com/paper/2412.01427/) (2024)
- [MB-TaylorFormer V2: Improved Multi-branch Linear Transformer Expanded by Taylor Formula for Image Restoration](https://mlsanity.com/paper/2501.04486/) (2025)
- [DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models](https://mlsanity.com/paper/2407.01519/) (2024)
- [PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions](https://mlsanity.com/paper/2409.15278/) (2024)
- [Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model](https://mlsanity.com/paper/2410.23905/) (2024)
- [Rethinking Diffusion Model for Multi-Contrast MRI Super-Resolution](https://mlsanity.com/paper/2404.04785/) (2024)
- [CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation](https://mlsanity.com/paper/2310.01407/) (2023)
- [NTIRE 2024 Challenge on Low Light Image Enhancement: Methods and Results](https://mlsanity.com/paper/2404.14248/) (2024)
- [Visual-Instructed Degradation Diffusion for All-in-One Image Restoration](https://mlsanity.com/paper/2506.16960/) (2025)
- [LLMGA: Multimodal Large Language Model based Generation Assistant](https://mlsanity.com/paper/2311.16500/) (2023)

## Abstract (arXiv)

Diffusion model (DM) has achieved SOTA performance by modeling the image synthesis process into a sequential application of a denoising network. However, different from image synthesis, image restoration (IR) has a strong constraint to generate results in accordance with ground-truth. Thus, for IR, traditional DMs running massive iterations on a large model to estimate whole images or feature maps is inefficient. To address this issue, we propose an efficient DM for IR (DiffIR), which consists of a compact IR prior extraction network (CPEN), dynamic IR transformer (DIRformer), and denoising network. Specifically, DiffIR has two training stages: pretraining and training DM. In pretraining, we input ground-truth images into CPENS1 to capture a compact IR prior representation (IPR) to guide DIRformer. In the second stage, we train the DM to directly estimate the same IRP as pretrained CPENS1 only using LQ images. We observe that since the IPR is only a compact vector, DiffIR can use fewer iterations than traditional DM to obtain accurate estimations and generate more stable and realistic results. Since the iterations are few, our DiffIR can adopt a joint optimization of CPENS2, DIRformer, and denoising network, which can further reduce the estimation error influence. We conduct extensive experiments on several IR tasks and achieve SOTA performance while consuming less computational costs. Code is available at urlhttps://github.com/Zj-BinXia/DiffIR.