mlsanityData to 14 Oct 2026

Upcycling Text-to-Image Diffusion Models for Multi-Task Capabilities

arXiv 2503.11905 ↗First posted 14 Mar 2025cs.CV cs.AIcited by 2

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-to-image synthesis has witnessed remarkable advancements in recent years. Many attempts have been made to adopt text-to-image models to support multiple tasks. However, existing approaches typically require resource-intensive re-training or additional parameters to accommodate for the new tasks, which makes the model inefficient for on-device deployment. We propose Multi-Task Upcycling (MTU), a simple yet effective recipe that extends the capabilities of a pre-trained text-to-image diffusion model to support a variety of image-to-image generation tasks. MTU replaces Feed-Forward Network (FFN) layers in the diffusion model with smaller FFNs, referred to as experts, and combines them with a dynamic routing mechanism. To the best of our knowledge, MTU is the first multi-task diffusion modeling approach that seamlessly blends multi-tasking with on-device compatibility, by mitigating the issue of parameter inflation. We show that the performance of MTU is on par with the single-task fine-tuned diffusion models across several tasks including image editing, super-resolution, and inpainting, while maintaining similar latency and computational load (GFLOPs) as the single-task fine-tuned models.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 46 of its 54 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

46 of 54 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023MobileDiffusion: Instant Text-to-Image Generation on Mobile Devices28 Nov
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  10. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  12. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
30 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  3. 2023One-step Diffusion with Distribution Matching Distillationcited by 1,019
  4. 2021Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Datacited by 970
  5. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  6. 2022On Distillation of Guided Diffusion Modelscited by 725
  7. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  8. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  9. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 273
  10. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillationcited by 266
  11. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  12. 2023Any-to-Any Generation via Composable Diffusioncited by 204
  13. 2023Structural Pruning for Diffusion Modelscited by 195
  14. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 187
  15. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  16. 2023PTQD: Accurate Post-Training Quantization for Diffusion Modelscited by 170
  17. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  18. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 132
  19. 2013Stabilitycited by 120
  20. 2024Distilling Diffusion Models into Conditional GANscited by 78
  21. 2024Diffusion Models, Image Super-Resolution And Everything: A Surveycited by 64
  22. 2024QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuningcited by 49
  23. 2024Paint by Inpaint: Learning to Add Image Objects by Removing Them Firstcited by 39
  24. 2024SlimFlow: Training Smaller One-Step Diffusion Models with Rectified Flowcited by 30
  25. 2024You Only Need One Step: Fast Super-Resolution with Stable Diffusion via Scale Distillationcited by 24
  26. 2024DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecturecited by 22
  27. 2024DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Datacited by 18
  28. 2024EdgeFusion: On-Device Text-to-Image Generationcited by 12
  29. 2023Lightweight Diffusion Models with Distillation-Based Block Neural Architecture Searchcited by 4
  30. 2024Multi-Task Label Discovery via Hierarchical Task Tokens for Partially Annotated Dense Predictionscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025AortaDiff: A Unified Multitask Diffusion Framework For Contrast-Free AAA Imagingcited by 1
  2. 2026Semantic-level Backdoor Attack against Text-to-Image Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023CycleNet: Rethinking Cycle Consistency in Text-Guided Diffusion for Image Manipulation
  2. 2023PanGu-Draw: Advancing Resource-Efficient Text-to-Image Synthesis with Time-Decoupled Training and Reusable Coop-Diffusion
  3. 2026DreamLite: A Lightweight On-Device Unified Model for Image Generation and Editing
  4. 2024Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Models
  5. 2023SpeedUpNet: A Plug-and-Play Adapter Network for Accelerating Text-to-Image Diffusion Models
  6. 2025UniVG: A Generalist Diffusion Model for Unified Image Generation and Editing
  7. 2025Reusing Computation in Text-to-Image Diffusion for Efficient Generation of Image Sets
  8. 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models