# SPIRE: Semantic Prompt-Driven Image Restoration

arXiv: https://arxiv.org/abs/2312.11595 · HTML: https://mlsanity.com/paper/2312.11595/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 74 of its 85 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 30 of 74)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)

## Built on by (most cited first, 24 of 24)

- [SleeperMark: Towards Robust Watermark against Fine-Tuning Text-to-image Diffusion Models](https://mlsanity.com/paper/2412.04852/) (2024)
- [Bigger is not Always Better: Scaling Properties of Latent Diffusion Models](https://mlsanity.com/paper/2404.01367/) (2024)
- [Edit Away and My Face Will not Stay: Personal Biometric Defense against Malicious Generative Editing](https://mlsanity.com/paper/2411.16832/) (2024)
- [The Power of Context: How Multimodality Improves Image Super-Resolution](https://mlsanity.com/paper/2503.14503/) (2025)
- [Instruct-IPT: All-in-One Image Processing Transformer via Weight Modulation](https://mlsanity.com/paper/2407.00676/) (2024)
- [ControlFusion: A Controllable Image Fusion Framework with Language-Vision Degradation Prompts](https://mlsanity.com/paper/2503.23356/) (2025)
- [MP-HSIR: A Multi-Prompt Framework for Universal Hyperspectral Image Restoration](https://mlsanity.com/paper/2503.09131/) (2025)
- [Uni-DocDiff: A Unified Document Restoration Model Based on Diffusion](https://mlsanity.com/paper/2508.04055/) (2025)
- [SparkVSR: Interactive Video Super-Resolution via Sparse Keyframe Propagation](https://mlsanity.com/paper/2603.16864/) (2026)
- [HFMF: Hierarchical Fusion Meets Multi-Stream Models for Deepfake Detection](https://mlsanity.com/paper/2501.05631/) (2025)
- [TPGDiff: Hierarchical Triple-Prior Guided Diffusion for Image Restoration](https://mlsanity.com/paper/2601.20306/) (2026)
- [Bridging Information Asymmetry: A Hierarchical Framework for Deterministic Blind Face Restoration](https://mlsanity.com/paper/2601.19506/) (2026)
- [Task-Guided Prompting for Unified Remote Sensing Image Restoration](https://mlsanity.com/paper/2604.02742/) (2026)
- [DeeDSR: Towards Real-World Image Super-Resolution via Degradation-Aware Stable Diffusion](https://mlsanity.com/paper/2404.00661/) (2024)
- [DifuzCam: Replacing Camera Lens with a Mask and a Diffusion Model](https://mlsanity.com/paper/2408.07541/) (2024)
- [InstructRestore: Region-Customized Image Restoration with Human Instructions](https://mlsanity.com/paper/2503.24357/) (2025)
- [TextSR: Diffusion Super-Resolution with Multilingual OCR Guidance](https://mlsanity.com/paper/2505.23119/) (2025)
- [T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs](https://mlsanity.com/paper/2511.16107/) (2025)
- [PixIE: Prompted Pixel-Space Low-Light Image Enhancement](https://mlsanity.com/paper/2605.23531/) (2026)
- [Referring Flexible Image Restoration](https://mlsanity.com/paper/2404.10342/) (2024)
- [Learning Ordinal Degradation Representations with Textual Priors for Diffusion-Based Blind Image Super-Resolution](https://mlsanity.com/paper/2512.10340/) (2025)
- [Generative Manifold Distillation: Aligning Restoration Trajectories with Natural Image Prior](https://mlsanity.com/paper/2512.11121/) (2025)
- [Beyond Fixed Inference: Quantitative Flow Matching for Adaptive Image Denoising](https://mlsanity.com/paper/2604.02392/) (2026)
- [MagnifiQ: Patch-aware Text Guided Progressive Upscaling for High-Resolution Image Restoration](https://mlsanity.com/paper/2608.14543/) (2026)

## Abstract (arXiv)

Text-driven diffusion models have become increasingly popular for various image editing tasks, including inpainting, stylization, and object replacement. However, it still remains an open research problem to adopt this language-vision paradigm for more fine-level image processing tasks, such as denoising, super-resolution, deblurring, and compression artifact removal. In this paper, we develop SPIRE, a Semantic and restoration Prompt-driven Image Restoration framework that leverages natural language as a user-friendly interface to control the image restoration process. We consider the capacity of prompt information in two dimensions. First, we use content-related prompts to enhance the semantic alignment, effectively alleviating identity ambiguity in the restoration outcomes. Second, our approach is the first framework that supports fine-level instruction through language-based quantitative specification of the restoration strength, without the need for explicit task-specific design. In addition, we introduce a novel fusion mechanism that augments the existing ControlNet architecture by learning to rescale the generative prior, thereby achieving better restoration fidelity. Our extensive experiments demonstrate the superior restoration performance of SPIRE compared to the state of the arts, alongside offering the flexibility of text-based control over the restoration effects.