# MoE-DiffIR: Task-customized Diffusion Priors for Universal Compressed Image Restoration

arXiv: https://arxiv.org/abs/2407.10833 · HTML: https://mlsanity.com/paper/2407.10833/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 34 of its 89 references that are in the map.
- First posted on arXiv 15 Jul 2024.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)
- Mixture-of-experts / [Vision mixture-of-experts](https://mlsanity.com/moe/vision-mixture-of-experts/)

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model](https://mlsanity.com/paper/2212.00490/) (2022)
- [Exploiting Diffusion Prior for Real-World Image Super-Resolution](https://mlsanity.com/paper/2305.07015/) (2023)
- [Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16322/) (2023)
- [High-Fidelity Generative Image Compression](https://mlsanity.com/paper/2006.09965/) (2020)
- [Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild](https://mlsanity.com/paper/2401.13627/) (2024)
- [DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior](https://mlsanity.com/paper/2308.15070/) (2023)
- [DiffIR: Efficient Diffusion Model for Image Restoration](https://mlsanity.com/paper/2303.09472/) (2023)
- [Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization](https://mlsanity.com/paper/2308.14469/) (2023)
- [From Sparse to Soft Mixtures of Experts](https://mlsanity.com/paper/2308.00951/) (2023)
- [GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium](https://mlsanity.com/paper/1706.08500/) (2017)
- [Image Restoration with Mean-Reverting Stochastic Differential Equations](https://mlsanity.com/paper/2301.11699/) (2023)
- [Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models](https://mlsanity.com/paper/2307.00619/) (2023)
- [PromptIR: Prompting for All-in-One Blind Image Restoration](https://mlsanity.com/paper/2306.13090/) (2023)
- [Controlling Vision-Language Models for Multi-Task Image Restoration](https://mlsanity.com/paper/2310.01018/) (2023)
- [Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2308.09388/) (2023)
- [AutoDIR: Automatic All-in-One Image Restoration with Latent Diffusion](https://mlsanity.com/paper/2310.10123/) (2023)
- [ProRes: Exploring Degradation-aware Visual Prompt for Universal Image Restoration](https://mlsanity.com/paper/2306.13653/) (2023)
- [Prompt-tuning latent diffusion models for inverse problems](https://mlsanity.com/paper/2310.01110/) (2023)
- [HAT: Hybrid Attention Transformer for Image Restoration](https://mlsanity.com/paper/2309.05239/) (2023)
- [Diffusion Models, Image Super-Resolution And Everything: A Survey](https://mlsanity.com/paper/2401.00736/) (2024)
- [Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration](https://mlsanity.com/paper/2312.02918/) (2023)

## Built on by (most cited first, 15 of 15)

- [Beyond Pixels: Text Enhances Generalization in Real-World Image Restoration](https://mlsanity.com/paper/2412.00878/) (2024)
- [Hybrid Agents for Image Restoration](https://mlsanity.com/paper/2503.10120/) (2025)
- [MambaCSR: Dual-Interleaved Scanning for Compressed Image Super-Resolution With SSMs](https://mlsanity.com/paper/2408.11758/) (2024)
- [NTIRE 2025 Challenge on Short-form UGC Video Quality Assessment and Enhancement: KwaiSR Dataset and Study](https://mlsanity.com/paper/2504.15003/) (2025)
- [Acquire and then Adapt: Squeezing out Text-to-Image Model for Image Restoration](https://mlsanity.com/paper/2504.15159/) (2025)
- [TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation](https://mlsanity.com/paper/2412.10275/) (2024)
- [M2Restore: Mixture-of-Experts-based Mamba-CNN Fusion Framework for All-in-One Image Restoration](https://mlsanity.com/paper/2506.07814/) (2025)
- [Adaptive Blind All-in-One Image Restoration](https://mlsanity.com/paper/2411.18412/) (2024)
- [Towards Unified Image Deblurring using a Mixture-of-Experts Decoder](https://mlsanity.com/paper/2508.06228/) (2025)
- [LossAgent: Towards Any Optimization Objectives for Image Processing with LLM Agents](https://mlsanity.com/paper/2412.04090/) (2024)
- [Why Compress What You Can Generate? When GPT-4o Generation Ushers in Image Compression Fields](https://mlsanity.com/paper/2504.21814/) (2025)
- [EndoIR: Degradation-Agnostic All-in-One Endoscopic Image Restoration via Noise-Aware Routing Diffusion](https://mlsanity.com/paper/2511.05873/) (2025)
- [Test-Time Preference Optimization for Image Restoration](https://mlsanity.com/paper/2511.19169/) (2025)
- [A Noise Constrained Diffusion (NC-Diffusion) Framework for High Fidelity Image Compression](https://mlsanity.com/paper/2604.06568/) (2026)
- [Image Quality Assessment for Machines: Paradigm, Large-scale Database, and Models](https://mlsanity.com/paper/2508.19850/) (2025)

## Abstract (arXiv)

We present MoE-DiffIR, an innovative universal compressed image restoration (CIR) method with task-customized diffusion priors. This intends to handle two pivotal challenges in the existing CIR methods: (i) lacking adaptability and universality for different image codecs, e.g., JPEG and WebP; (ii) poor texture generation capability, particularly at low bitrates. Specifically, our MoE-DiffIR develops the powerful mixture-of-experts (MoE) prompt module, where some basic prompts cooperate to excavate the task-customized diffusion priors from Stable Diffusion (SD) for each compression task. Moreover, the degradation-aware routing mechanism is proposed to enable the flexible assignment of basic prompts. To activate and reuse the cross-modality generation prior of SD, we design the visual-to-text adapter for MoE-DiffIR, which aims to adapt the embedding of low-quality images from the visual domain to the textual domain as the textual guidance for SD, enabling more consistent and reasonable texture generation. We also construct one comprehensive benchmark dataset for universal CIR, covering 21 types of degradations from 7 popular traditional and learned codecs. Extensive experiments on universal CIR have demonstrated the excellent robustness and texture restoration capability of our proposed MoE-DiffIR. The project can be found at https://renyulin-f.github.io/MoE-DiffIR.github.io/.