mlsanityData to 14 Oct 2026

Controlling Vision-Language Models for Multi-Task Image Restoration

arXiv 2310.01018 ↗First posted 2 Oct 2023cs.CVcited by 110

On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment

Abstract (arXiv)

Vision-language models such as CLIP have shown great impact on diverse downstream tasks for zero-shot or label-free predictions. However, when it comes to low-level vision such as image restoration their performance deteriorates dramatically due to corrupted inputs. In this paper, we present a degradation-aware vision-language model (DA-CLIP) to better transfer pretrained vision-language models to low-level vision tasks as a multi-task framework for image restoration. More specifically, DA-CLIP trains an additional controller that adapts the fixed CLIP image encoder to predict high-quality feature embeddings. By integrating the embedding into an image restoration network via cross-attention, we are able to pilot the model to learn a high-fidelity image reconstruction. The controller itself will also output a degradation feature that matches the real corruptions of the input, yielding a natural classifier for different degradation types. In addition, we construct a mixed degradation dataset with synthetic captions for DA-CLIP training. Our approach advances state-of-the-art performance on both degradation-specific and unified image restoration tasks, showing a promising direction of prompting image restoration with large-scale pretrained vision-language models. Our code is available at https://github.com/Algolzw/daclip-uir.

Key facts. Cited by 110 papers in the map (data through 14 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 59 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

46 of 59 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  12. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
34 more
  1. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,627
  2. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  3. 2016Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoisingcited by 1,358
  4. 2019Contrastive Multiview Codingcited by 1,301
  5. 2021Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Datacited by 970
  6. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  7. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  8. 2021Multi-Stage Progressive Image Restorationcited by 816
  9. 2016Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurringcited by 727
  10. 2018Deep Retinex Decomposition for Low-Light Enhancementcited by 723
  11. 2022Simple Baselines for Image Restorationcited by 708
  12. 2021Designing a Practical Degradation Model for Deep Blind Image Super-Resolutioncited by 529
  13. 2023Exploiting Diffusion Prior for Real-World Image Super-Resolutioncited by 489
  14. 2019EnlightenGAN: Deep Light Enhancement without Paired Supervisioncited by 466
  15. 2017DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networkscited by 367
  16. 2019DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Bettercited by 362
  17. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 362
  18. 2016Deep Joint Rain Detection and Removal from a Single Imagecited by 352
  19. 2020Learning Enriched Features for Real Image Restoration and Enhancementcited by 312
  20. 2019FFA-Net: Feature Fusion Attention Network for Single Image Dehazingcited by 307
  21. 2022MAXIM: Multi-Axis MLP for Image Processingcited by 302
  22. 2019Progressive Image Deraining Networks: A Better and Simpler Baselinecited by 272
  23. 2022Vision Transformers for Single Image Dehazingcited by 260
  24. 2023DiffBIR: Towards Blind Image Restoration with Generative Diffusion Priorcited by 257
  25. 2019GridDehazeNet: Attention-Based Multi-Scale Network for Image Dehazingcited by 227
  26. 2017Attentive Generative Adversarial Network for Raindrop Removal from a Single Imagecited by 204
  27. 2017DesnowNet: Context-Aware Deep Network for Snow Removalcited by 200
  28. 2023Image Restoration with Mean-Reverting Stochastic Differential Equationscited by 183
  29. 2023PromptIR: Prompting for All-in-One Blind Image Restorationcited by 114
  30. 2018Gated Context Aggregation Network for Image Dehazing and Derainingcited by 89
  31. 2023Refusion: Enabling Large-Size Realistic Image Restoration with Latent-Space Diffusion Modelscited by 86
  32. 2019Single Image Deraining: A Comprehensive Benchmark Analysiscited by 84
  33. 2021Finding Discriminative Filters for Specific Degradations in Blind Super-Resolutioncited by 26
  34. 2022N"UWA-LIP: Language Guided Image Inpainting with Defect-free VQGANcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 16; 2025: 39; 2026: 50.

110 papers.

Sort:

By area:

  1. 2024Complexity Experts are Task-Discriminative Learners for Any Image Restorationcited by 77
  2. 2023CoSeR: Bridging Image and Language for Cognitive Super-Resolutioncited by 66
  3. 2024Perceive-IR: Learning to Perceive Degradation Better for All-in-One Image Restorationcited by 62
  4. 2024A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trendscited by 60
  5. 2023Prompt-In-Prompt Learning for Universal Image Restorationcited by 43
  6. 2024An Intelligent Agentic System for Complex Image Restoration Problemscited by 43
  7. 2025Vision-Language Gradient Descent-driven All-in-One Deep Unfolding Networkscited by 37
  8. 2025JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restorationcited by 34
  9. 2023Regularization by Texts for Latent Diffusion Inverse Solverscited by 32
  10. 2023Textual Prompt Guided Image Restorationcited by 27
  11. 2023SPIRE: Semantic Prompt-Driven Image Restorationcited by 24
  12. 2024Degradation-Aware Residual-Conditioned Optimal Transport for Unified Image Restorationcited by 21
38 more
  1. 2025Toward Generalizable Forgery Detection and Reasoningcited by 20
  2. 2024Multi-Expert Adaptive Selection: Task-Balancing for All-in-One Image Restorationcited by 18
  3. 2024Diff-Restorer: Unleashing Visual Prompts for Diffusion-based Universal Image Restorationcited by 15
  4. 2024MoE-DiffIR: Task-customized Diffusion Priors for Universal Compressed Image Restorationcited by 15
  5. 2025Dual Prompting Image Restoration with Diffusion Transformerscited by 15
  6. 2025Residual Diffusion Bridge Model for Image Restorationcited by 15
  7. 2026Learning Continuous Wasserstein Barycenter Space for Generalized All-in-One Image Restorationcited by 15
  8. 2024DP-IQA: Utilizing Diffusion Prior for Blind Image Quality Assessment in the Wildcited by 14
  9. 2025MMAIF: Multi-task and Multi-degradation All-in-One for Image Fusion with Language Guidancecited by 9
  10. 2025ControlFusion: A Controllable Image Fusion Framework with Language-Vision Degradation Promptscited by 9
  11. 2025LD-RPS: Zero-Shot Unified Image Restoration via Latent Diffusion Recurrent Posterior Samplingcited by 9
  12. 2024LLMRA: Multi-modal Large Language Model based Restoration Assistantcited by 8
  13. 2025DSwinIR: Rethinking Window-based Attention for Image Restorationcited by 7
  14. 2025M2Restore: Mixture-of-Experts-based Mamba-CNN Fusion Framework for All-in-One Image Restorationcited by 7
  15. 2025Beyond Degradation Redundancy: Contrastive Prompt Learning for All-in-One Image Restorationcited by 6
  16. 2025Restore-R1: Efficient Image Restoration Agents via Reinforcement Learning with Multimodal LLM Perceptual Feedbackcited by 6
  17. 2024Learning Efficient and Effective Trajectories for Differential Equation-based Image Restorationcited by 5
  18. 2025MaRS: A Fast Sampler for Mean Reverting Diffusion based on ODE and SDE Solverscited by 5
  19. 2025DGSolver: Diffusion Generalist Solver with Universal Posterior Sampling for Image Restorationcited by 5
  20. 2025UniRes: Universal Image Restoration for Complex Degradationscited by 5
  21. 2026TIR-Agent: Training an Explorative and Efficient Agent for Image Restorationcited by 5
  22. 2026Decoupled Residual Denoising Diffusion Models for Unified and Data Efficient Image-to-Image Translationcited by 5
  23. 2024Few-Shot Image Quality Assessment via Adaptation of Vision-Language Modelscited by 4
  24. 2025HazeFlow: Revisit Haze Physical Model as ODE and Non-Homogeneous Haze Generation for Real-World Dehazingcited by 4
  25. 2025PhyDAE: Physics-Guided Degradation-Adaptive Experts for All-in-One Remote Sensing Image Restorationcited by 4
  26. 2025Vision-Language Model Guided Image Restorationcited by 4
  27. 2026OPERA: An Agent for Image Restoration with End-to-End Joint Planning-Execution Optimizationcited by 4
  28. 2026Self-Evolving Agentic Image Restoration via Deliberate Planning and Intuitive Executioncited by 4
  29. 2024Review Learning: Advancing All-in-One Ultra-High-Definition Image Restoration Training Methodcited by 3
  30. 2024ReviveDiff: A Universal Diffusion Model for Restoring Images in Adverse Weather Conditionscited by 3
  31. 2025BaryIR: Learning Multi-Source Unified Representation in Continuous Barycenter Space for Generalizable All-in-One Image Restorationcited by 3
  32. 2025Demystifying the Visual Quality Paradox in Multimodal Large Language Modelscited by 3
  33. 2025Elucidating and Endowing the Diffusion Training Paradigm for General Image Restorationcited by 3
  34. 2025AWM-Fuse: Multi-Modality Image Fusion for Adverse Weather via Global and Local Text Perceptioncited by 3
  35. 2025Enhancing Infrared Vision: Progressive Prompt Fusion Network and Benchmarkcited by 3
  36. 2026Bridging Information Asymmetry: A Hierarchical Framework for Deterministic Blind Face Restorationcited by 3
  37. 2026V-Bridge: Bridging Video Generative Priors to Versatile Few-shot Image Restorationcited by 3
  38. 2026Restore, Assess, Repeat: A Unified Framework for Iterative Image Restorationcited by 3

50 most cited of 110 All 110 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding Degradation with Vision Language Model
  2. 2024Photo-Realistic Image Restoration in the Wild with Controlled Vision-Language Models
  3. 2024Restore Anything Model via Efficient Degradation Adaptation
  4. 2025Degradation-Aware Image Enhancement via Vision-Language Classification
  5. 2026DPC-Net: Dual-Prior Collaborative Network for All-in-One Image Restoration
  6. 2024Chain-of-Restoration: Multi-Task Image Restoration Models are Zero-Shot Step-by-Step Universal Image Restorers
  7. 2026Edit2Restore:Few-Shot Image Restoration via Parameter-Efficient Adaptation of Pre-trained Editing Models
  8. 2022Learning to Decompose Visual Features with Latent Textual Prompts