# Photo-Realistic Image Restoration in the Wild with Controlled Vision-Language Models

arXiv: https://arxiv.org/abs/2404.09732 · HTML: https://mlsanity.com/paper/2404.09732/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 14 of its 71 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Image editing and quality assessment](https://mlsanity.com/mllm/image-editing-and-quality-assessment/)
- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 14 of 14)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Deep Retinex Decomposition for Low-Light Enhancement](https://mlsanity.com/paper/1808.04560/) (2018)
- [Exploiting Diffusion Prior for Real-World Image Super-Resolution](https://mlsanity.com/paper/2305.07015/) (2023)
- [SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution](https://mlsanity.com/paper/2311.16518/) (2023)
- [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023)
- [Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild](https://mlsanity.com/paper/2401.13627/) (2024)
- [DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior](https://mlsanity.com/paper/2308.15070/) (2023)
- [CoSeR: Bridging Image and Language for Cognitive Super-Resolution](https://mlsanity.com/paper/2311.16512/) (2023)
- [Denoising Diffusion Probabilistic Models for Robust Image Super-Resolution in the Wild](https://mlsanity.com/paper/2302.07864/) (2023)
- [Image Restoration Through Generalized Ornstein-Uhlenbeck Bridge](https://mlsanity.com/paper/2312.10299/) (2023)
- [Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning](https://mlsanity.com/paper/2402.04080/) (2024)

## Built on by (most cited first, 19 of 19)

- [A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trends](https://mlsanity.com/paper/2410.15067/) (2024)
- [Vision-Language Gradient Descent-driven All-in-One Deep Unfolding Networks](https://mlsanity.com/paper/2503.16930/) (2025)
- [Unveiling Hidden Details: A RAW Data-Enhanced Paradigm for Real-World Super-Resolution](https://mlsanity.com/paper/2411.10798/) (2024)
- [Beyond Pixels: Text Enhances Generalization in Real-World Image Restoration](https://mlsanity.com/paper/2412.00878/) (2024)
- [Taming Diffusion Models for Image Restoration: A Review](https://mlsanity.com/paper/2409.10353/) (2024)
- [HazeCLIP: Towards Language Guided Real-World Image Dehazing](https://mlsanity.com/paper/2407.13719/) (2024)
- [MP-HSIR: A Multi-Prompt Framework for Universal Hyperspectral Image Restoration](https://mlsanity.com/paper/2503.09131/) (2025)
- [MaRS: A Fast Sampler for Mean Reverting Diffusion based on ODE and SDE Solvers](https://mlsanity.com/paper/2502.07856/) (2025)
- [UniRes: Universal Image Restoration for Complex Degradations](https://mlsanity.com/paper/2506.05599/) (2025)
- [Vision-Language Model Guided Image Restoration](https://mlsanity.com/paper/2512.17292/) (2025)
- [Efficient Image Restoration with State-Dependent Forward Diffusion](https://mlsanity.com/paper/2505.16733/) (2025)
- [Enhancing Diffusion-based Restoration Models via Difficulty-Adaptive Reinforcement Learning with IQA Reward](https://mlsanity.com/paper/2511.01645/) (2025)
- [DINOLight: Robust Ambient Light Normalization with Self-supervised Visual Prior Integration](https://mlsanity.com/paper/2603.12579/) (2026)
- [PixRestore: Unified Image Restoration via Pixel Diffusion Transformer](https://mlsanity.com/paper/2608.16793/) (2026)
- [Consistent Diffusion: Denoising Diffusion Model with Data-Consistent Training for Image Restoration](https://mlsanity.com/paper/2412.12550/) (2024)
- [GGT-100K: Generative Ground Truth for Generalizable Real-World Image Restoration](https://mlsanity.com/paper/2605.31039/) (2026)
- [Boosting Fidelity for Pre-Trained-Diffusion-Based Low-Light Image Enhancement via Condition Refinement](https://mlsanity.com/paper/2510.17105/) (2025)
- [Seeing Through the PRISM: Compound & Controllable Restoration of Scientific Images](https://mlsanity.com/paper/2603.14151/) (2026)
- [HarnessIR: Harnessing Multimodal Foundation Models for Universal Real-World Image Restoration](https://mlsanity.com/paper/2610.10133/) (2026)

## Abstract (arXiv)

Though diffusion models have been successfully applied to various image restoration (IR) tasks, their performance is sensitive to the choice of training datasets. Typically, diffusion models trained in specific datasets fail to recover images that have out-of-distribution degradations. To address this problem, this work leverages a capable vision-language model and a synthetic degradation pipeline to learn image restoration in the wild (wild IR). More specifically, all low-quality images are simulated with a synthetic degradation pipeline that contains multiple common degradations such as blur, resize, noise, and JPEG compression. Then we introduce robust training for a degradation-aware CLIP model to extract enriched image content features to assist high-quality image restoration. Our base diffusion model is the image restoration SDE (IR-SDE). Built upon it, we further present a posterior sampling strategy for fast noise-free image generation. We evaluate our model on both synthetic and real-world degradation datasets. Moreover, experiments on the unified image restoration task illustrate that the proposed posterior sampling improves image generation quality for various degradations.