# Visual-Instructed Degradation Diffusion for All-in-One Image Restoration

arXiv: https://arxiv.org/abs/2506.16960 · HTML: https://mlsanity.com/paper/2506.16960/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 108 of its 147 references that are in the map.
- First posted on arXiv 20 Jun 2025.

## Builds on (most cited first, 30 of 108)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)

## Built on by (most cited first, 26 of 26)

- [FAPE-IR: Frequency-Aware Planning and Execution Framework for All-in-One Image Restoration](https://mlsanity.com/paper/2511.14099/) (2025)
- [Generative Refocusing: Flexible Defocus Control from a Single Image](https://mlsanity.com/paper/2512.16923/) (2025)
- [TPGDiff: Hierarchical Triple-Prior Guided Diffusion for Image Restoration](https://mlsanity.com/paper/2601.20306/) (2026)
- [BIR-Adapter: A parameter-efficient diffusion adapter for blind image restoration](https://mlsanity.com/paper/2509.06904/) (2025)
- [EndoIR: Degradation-Agnostic All-in-One Endoscopic Image Restoration via Noise-Aware Routing Diffusion](https://mlsanity.com/paper/2511.05873/) (2025)
- [Learning Adaptive Dynamical Features via Multi-τ Liquid-Mamba for All-in-one Image Restoration](https://mlsanity.com/paper/2606.22801/) (2026)
- [Clear Roads, Clear Vision: Advancements in Multi-Weather Restoration for Smart Transportation](https://mlsanity.com/paper/2510.09228/) (2025)
- [Degradation-Aware Metric Prompting for Hyperspectral Image Restoration](https://mlsanity.com/paper/2512.20251/) (2025)
- [Unifying Heterogeneous Degradations: Uncertainty-Aware Diffusion Bridge Model for All-in-One Image Restoration](https://mlsanity.com/paper/2601.21592/) (2026)
- [Causal-AgentIR: Self-Evolving Causal Memory for Adaptive Image Restoration Agents](https://mlsanity.com/paper/2607.21125/) (2026)
- [Bend the Basics: Degradation-Aware Deformable Tokenization for All-in-One Image Restoration](https://mlsanity.com/paper/2608.06832/) (2026)
- [HNDiff: Haze-Noise Diffusion for Image Dehazing](https://mlsanity.com/paper/2608.10995/) (2026)
- [Learning to Restore Multi-Degraded Images via Ingredient Decoupling and Task-Aware Path Adaptation](https://mlsanity.com/paper/2511.04920/) (2025)
- [Physically Interpretable Multi-Degradation Image Restoration via Deep Unfolding and Explainable Convolution](https://mlsanity.com/paper/2511.10166/) (2025)
- [Unleashing Degradation-Carrying Features in Symmetric U-Net: Simpler and Stronger Baselines for All-in-One Image Restoration](https://mlsanity.com/paper/2512.10581/) (2025)
- [From Physical Degradation Models to Task-Aware All-in-One Image Restoration](https://mlsanity.com/paper/2601.10192/) (2026)
- [UnSCAR: Universal, Scalable, Controllable, and Adaptable Image Restoration](https://mlsanity.com/paper/2603.07406/) (2026)
- [EvoIR-Agent: Self-Evolving Image Restoration Agentic System via Experience-Driven Learning](https://mlsanity.com/paper/2605.22208/) (2026)
- [CV-Arena: An Open Benchmark for Instructional Computer Vision Problem Solving with Human-AI Collaborative Preferences](https://mlsanity.com/paper/2606.00931/) (2026)
- [RQUL-UIE: Revitalizing Quality-Unstable Labels for Underwater Image Enhancement via In-Dataset Self-Supervision](https://mlsanity.com/paper/2606.06176/) (2026)
- [Fusing Transferred Priors and Physics-based Decomposition for Underwater Image Enhancement](https://mlsanity.com/paper/2606.15648/) (2026)
- [Beyond Uniform Restoration: Empowering All-in-One Restoration with Pixel-Level Multimodal Guidance](https://mlsanity.com/paper/2608.09482/) (2026)
- [Learning to Restore More: Continual Capability Expansion for Pretrained Image Restoration Models](https://mlsanity.com/paper/2608.30305/) (2026)
- [GraLoD: Graphics-Inspired Continuous Level-of-Detail Learning for Image Restoration](https://mlsanity.com/paper/2609.16578/) (2026)
- [ImIR: Image-Instruction Tuning for All-in-One Image Restoration](https://mlsanity.com/paper/2609.25267/) (2026)
- [Geometric Similarity in VLM Low-Level Vision Representations](https://mlsanity.com/paper/2610.00848/) (2026)

## Abstract (arXiv)

Image restoration tasks like deblurring, denoising, and dehazing usually need distinct models for each degradation type, restricting their generalization in real-world scenarios with mixed or unknown degradations. In this work, we propose Defusion, a novel all-in-one image restoration framework that utilizes visual instruction-guided degradation diffusion. Unlike existing methods that rely on task-specific models or ambiguous text-based priors, Defusion constructs explicit visual instructions that align with the visual degradation patterns. These instructions are grounded by applying degradations to standardized visual elements, capturing intrinsic degradation features while agnostic to image semantics. Defusion then uses these visual instructions to guide a diffusion-based model that operates directly in the degradation space, where it reconstructs high-quality images by denoising the degradation effects with enhanced stability and generalizability. Comprehensive experiments demonstrate that Defusion outperforms state-of-the-art methods across diverse image restoration tasks, including complex and real-world degradations.