Leveraging Multimodal Large Language Models for All-in-One Image Restoration via a Mixture of Frequency Experts
On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment
Abstract (arXiv)
All-in-one image restoration seeks to recover clean images from inputs affected by diverse and unknown degradations using a unified framework. Recent methods have shown strong performance by identifying degradation characteristics to guide the restoration process. However, many of them treat degradations as discrete categories, which limits their ability to model the continuous relational structure that arises in composite degradations. To address this issue, we propose a multimodal large language model (MLLM)-guided image restoration framework that exploits multimodal embeddings as guidance for low-level restoration. Specifically, MLLM-derived features are injected into an encoder-decoder architecture through an MLLM-guided fusion block (MGFB) to enhance degradation-aware representations. In addition, we incorporate a mixture-of-frequency-experts (MoFE) module that adaptively combines frequency experts using MLLM-guided contextual cues. To further improve expert routing, we design an MLLM-guided router with a relational alignment loss that encourages routing patterns consistent with the embedding-space relationships of degraded inputs. Extensive experiments on multiple benchmarks show that the proposed method achieves strong performance across diverse restoration settings and establishes a new state of the art on the challenging CDD11 dataset, outperforming previous methods by up to 1.35 dB.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 62 of its 79 references that are in the map. First posted on arXiv 12 May 2026.
Builds on
62 of 79 references.
Closest ancestors · 2
Same direction, newest first
- 2024RestoreAgent: Autonomous Image Restoration Agent via Multimodal Large Language Models25 Jul
- 2023Controlling Vision-Language Models for Multi-Task Image Restoration2 Oct
Other references · 60
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2023GPT-4 Technical Reportcited by 18,396
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
- 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
38 more
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
- 2021SwinIR: Image Restoration Using Swin Transformercited by 1,887
- 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,627
- 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
- 2021Uformer: A General U-Shaped Transformer for Image Restorationcited by 866
- 2021Multi-Stage Progressive Image Restorationcited by 816
- 2016Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurringcited by 727
- 2018Deep Retinex Decomposition for Low-Light Enhancementcited by 723
- 2022Simple Baselines for Image Restorationcited by 708
- 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
- 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
- 2017Benchmarking Single Image Dehazing and Beyondcited by 447
- 2017Learning Deep CNN Denoiser Prior for Image Restorationcited by 408
- 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
- 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 347
- 2022How Do Vision Transformers Work?cited by 307
- 2018Toward Convolutional Blind Denoising of Real Photographscited by 274
- 2019Progressive Image Deraining Networks: A Better and Simpler Baselinecited by 272
- 2020Multi-Scale Progressive Fusion Network for Single Image Derainingcited by 260
- 2021TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditionscited by 244
- 2019Residual Non-local Attention Networks for Image Restorationcited by 236
- 2022Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Modelscited by 215
- 2020CycleISP: Real Image Restoration via Improved Data Synthesiscited by 213
- 2020Deblurring by Realistic Blurringcited by 188
- 2018Densely Connected Pyramid Dehazing Networkcited by 145
- 2024InstructIR: High-Quality Image Restoration Following Human Instructionscited by 140
- 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
- 2018Gated Fusion Network for Single Image Dehazingcited by 133
- 2024AdaIR: Adaptive All-in-One Image Restoration via Frequency Mining and Modulationcited by 111
- 2023AutoDIR: Automatic All-in-One Image Restoration with Latent Diffusioncited by 100
- 2024OneRestore: A Universal Restoration Framework for Composite Degradationcited by 92
- 2024Complexity Experts are Task-Discriminative Learners for Any Image Restorationcited by 77
- 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 77
- 2019Dual Residual Networks Leveraging the Potential of Paired Operations for Image Restorationcited by 73
- 2024DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curationcited by 65
- 2023GridFormer: Residual Dense Transformer with Grid Structure for Image Restoration in Adverse Weather Conditionscited by 62
- 2023Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restorationcited by 50
- 2023Neural Degradation Representation Learning for All-In-One Image Restorationcited by 46
50 most cited of 60 All 60 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024LLMRA: Multi-modal Large Language Model based Restoration Assistant
- 2025ClusIR: Towards Cluster-Guided All-in-One Image Restoration
- 2026M2IR: Proactive All-in-One Image Restoration via Mamba-style Modulation and Mixture-of-Experts
- 2025Restore-R1: Efficient Image Restoration Agents via Reinforcement Learning with Multimodal LLM Perceptual Feedback
- 2026UnSCAR: Universal, Scalable, Controllable, and Adaptable Image Restoration
- 2026Continuous Expert Assembly: Instance-Conditioned Low-Rank Residuals for All-in-One Image Restoration
- 2026DPC-Net: Dual-Prior Collaborative Network for All-in-One Image Restoration
- 2024Chain-of-Restoration: Multi-Task Image Restoration Models are Zero-Shot Step-by-Step Universal Image Restorers