# Improving Image Restoration through Removing Degradations in Textual Representations

arXiv: https://arxiv.org/abs/2312.17334 · HTML: https://mlsanity.com/paper/2312.17334/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 84 of its 122 references that are in the map.
- First posted on arXiv 28 Dec 2023.

## Builds on (most cited first, 30 of 84)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [Multi-Stage Progressive Image Restoration](https://mlsanity.com/paper/2102.02808/) (2021)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [Simple Baselines for Image Restoration](https://mlsanity.com/paper/2204.04676/) (2022)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [Exploiting Diffusion Prior for Real-World Image Super-Resolution](https://mlsanity.com/paper/2305.07015/) (2023)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [Plug-and-Play Image Restoration with Deep Denoiser Prior](https://mlsanity.com/paper/2008.13751/) (2020)
- [Benchmarking Single Image Dehazing and Beyond](https://mlsanity.com/paper/1712.04143/) (2017)
- [Activating More Pixels in Image Super-Resolution Transformer](https://mlsanity.com/paper/2205.04437/) (2022)
- [ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation](https://mlsanity.com/paper/2302.13848/) (2023)

## Built on by (most cited first, 30 of 36)

- [A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trends](https://mlsanity.com/paper/2410.15067/) (2024)
- [FoundIR: Unleashing Million-scale Training Data to Advance Foundation Models for Image Restoration](https://mlsanity.com/paper/2412.01427/) (2024)
- [FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models](https://mlsanity.com/paper/2401.15636/) (2024)
- [The Tenth NTIRE 2025 Image Denoising Challenge Report](https://mlsanity.com/paper/2504.12276/) (2025)
- [Deblur4DGS: 4D Gaussian Splatting from Blurry Monocular Video](https://mlsanity.com/paper/2412.06424/) (2024)
- [Multi-Expert Adaptive Selection: Task-Balancing for All-in-One Image Restoration](https://mlsanity.com/paper/2407.19139/) (2024)
- [AWRaCLe: All-Weather Image Restoration using Visual In-Context Learning](https://mlsanity.com/paper/2409.00263/) (2024)
- [Beyond Pixels: Text Enhances Generalization in Real-World Image Restoration](https://mlsanity.com/paper/2412.00878/) (2024)
- [Diff-Restorer: Unleashing Visual Prompts for Diffusion-based Universal Image Restoration](https://mlsanity.com/paper/2407.03636/) (2024)
- [VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models](https://mlsanity.com/paper/2403.05438/) (2024)
- [UniRestorer: Universal Image Restoration via Adaptively Estimating Image Degradation at Proper Granularity](https://mlsanity.com/paper/2412.20157/) (2024)
- [NTIRE 2025 Challenge on Efficient Burst HDR and Restoration: Datasets, Methods, and Results](https://mlsanity.com/paper/2505.12089/) (2025)
- [Exploiting Diffusion Prior for Real-World Image Dehazing with Unpaired Training](https://mlsanity.com/paper/2503.15017/) (2025)
- [Rethinking Transformer-Based Blind-Spot Network for Self-Supervised Image Denoising](https://mlsanity.com/paper/2404.07846/) (2024)
- [Instruct-IPT: All-in-One Image Processing Transformer via Weight Modulation](https://mlsanity.com/paper/2407.00676/) (2024)
- [PromptHSI: Universal Hyperspectral Image Restoration with Vision-Language Modulated Frequency Adaptation](https://mlsanity.com/paper/2411.15922/) (2024)
- [Beyond Degradation Redundancy: Contrastive Prompt Learning for All-in-One Image Restoration](https://mlsanity.com/paper/2504.09973/) (2025)
- [Robo-SGG: Exploiting Layout-Oriented Normalization and Restitution Can Improve Robust Scene Graph Generation](https://mlsanity.com/paper/2504.12606/) (2025)
- [Grounding Degradations in Natural Language for All-In-One Video Restoration](https://mlsanity.com/paper/2507.14851/) (2025)
- [Robust Adverse Weather Removal via Spectral-based Spatial Grouping](https://mlsanity.com/paper/2507.22498/) (2025)
- [AWM-Fuse: Multi-Modality Image Fusion for Adverse Weather via Global and Local Text Perception](https://mlsanity.com/paper/2508.16881/) (2025)
- [MasterWeaver: Taming Editability and Face Identity for Personalized Text-to-Image Generation](https://mlsanity.com/paper/2405.05806/) (2024)
- [Reblurring-Guided Single Image Defocus Deblurring: A Learning Framework with Misaligned Training Pairs](https://mlsanity.com/paper/2409.17792/) (2024)
- [Mixed Degradation Image Restoration via Local Dynamic Optimization and Conditional Embedding](https://mlsanity.com/paper/2411.16217/) (2024)
- [VL-UR: Vision-Language-guided Universal Restoration of Images Degraded by Adverse Weather Conditions](https://mlsanity.com/paper/2504.08219/) (2025)
- [All-in-One Image Restoration via Causal-Deconfounding Wavelet-Disentangled Prompt Network](https://mlsanity.com/paper/2603.03839/) (2026)
- [DINOLight: Robust Ambient Light Normalization with Self-supervised Visual Prior Integration](https://mlsanity.com/paper/2603.12579/) (2026)
- [Consistent Diffusion: Denoising Diffusion Model with Data-Consistent Training for Image Restoration](https://mlsanity.com/paper/2412.12550/) (2024)
- [Boosting All-in-One Image Restoration via Self-Improved Privilege Learning](https://mlsanity.com/paper/2505.24207/) (2025)
- [Degradation-Aware Prompt Learning with Cross-Modal Compensation for Adverse Weather Removal](https://mlsanity.com/paper/2608.06939/) (2026)

## Abstract (arXiv)

In this paper, we introduce a new perspective for improving image restoration by removing degradation in the textual representations of a given degraded image. Intuitively, restoration is much easier on text modality than image one. For example, it can be easily conducted by removing degradation-related words while keeping the content-aware words. Hence, we combine the advantages of images in detail description and ones of text in degradation removal to perform restoration. To address the cross-modal assistance, we propose to map the degraded images into textual representations for removing the degradations, and then convert the restored textual representations into a guidance image for assisting image restoration. In particular, We ingeniously embed an image-to-text mapper and text restoration module into CLIP-equipped text-to-image models to generate the guidance. Then, we adopt a simple coarse-to-fine approach to dynamically inject multi-scale information from guidance to image restoration networks. Extensive experiments are conducted on various image restoration tasks, including deblurring, dehazing, deraining, and denoising, and all-in-one image restoration. The results showcase that our method outperforms state-of-the-art ones across all these tasks. The codes and models are available at urlhttps://github.com/mrluin/TextualDegRemoval.