mlsanityData to 14 Oct 2026

Decouple-Then-Merge: Finetune Diffusion Models as Multi-Task Learning

arXiv 2410.06664 ↗First posted 9 Oct 2024cs.CV cs.AIcited by 7

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models are trained by learning a sequence of models that reverse each step of noise corruption. Typically, the model parameters are fully shared across multiple timesteps to enhance training efficiency. However, since the denoising tasks differ at each timestep, the gradients computed at different timesteps may conflict, potentially degrading the overall performance of image generation. To solve this issue, this work proposes a Decouple-then-Merge (DeMe) framework, which begins with a pretrained model and finetunes separate models tailored to specific timesteps. We introduce several improved techniques during the finetuning stage to promote effective knowledge sharing while minimizing training interference across timesteps. Finally, after finetuning, these separate models can be merged into a single model in the parameter space, ensuring efficient and practical inference. Experimental results show significant generation quality improvements upon 6 benchmarks including Stable Diffusion on COCO30K, ImageNet1K, PartiPrompts, and DDPM on LSUN Church, LSUN Bedroom, and CIFAR10. Code is available at hrefhttps://github.com/MqLeet/DeMeGitHub.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 59 of its 67 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

59 of 67 references.

Closest ancestors · 28

Same direction, newest first

  1. 2024Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selection23 Sep
  2. 2024Diffusion Model Patching via Mixture-of-Prompts28 May
  3. 2024Linear Combination of Saved Checkpoints Makes Consistency and Diffusion Models Better2 Apr
  4. 2024Denoising Task Difficulty-based Curriculum for Training Diffusion Models15 Mar
  5. 2024Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Experts14 Mar
  6. 2023Improving Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architectures14 Dec
  7. 2023Denoising Task Routing for Diffusion Models11 Oct
  8. 2023OMS-DPM: Optimizing the Model Schedule for Diffusion Probabilistic Models15 Jun
  9. 2023Multi-Architecture Multi-Expert Diffusion Models8 Jun
  10. 2023Addressing Negative Transfer in Diffusion Models1 Jun
  11. 2023Structural Pruning for Diffusion Models18 May
  12. 2023Efficient Diffusion Training via Min-SNR Weighting Strategy16 Mar
16 more
  1. 2023Q-Diffusion: Quantizing Diffusion Models8 Feb
  2. 2022Scalable Diffusion Models with Transformers19 Dec
  3. 2022Towards Practical Plug-and-Play Diffusion Models12 Dec
  4. 2022Post-training Quantization on Diffusion Models28 Nov
  5. 2022Diffusion Probabilistic Model Made Slim27 Nov
  6. 2022On Distillation of Guided Diffusion Models6 Oct
  7. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  8. 2022Perception Prioritized Training of Diffusion Models1 Apr
  9. 2022Pseudo Numerical Methods for Diffusion Models on Manifolds20 Feb
  10. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  11. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  12. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  13. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed7 Jan
  14. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  15. 2020Denoising Diffusion Implicit Models6 Oct
  16. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  10. 2022Video Diffusion Modelscited by 2,305
  11. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
19 more
  1. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  2. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,355
  3. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  4. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  6. 2022Editing Models with Task Arithmeticcited by 1,120
  7. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  8. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  9. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  10. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  11. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  12. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  13. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  14. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
  15. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  16. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
  17. 2022Protein structure generation via folding diffusioncited by 115
  18. 2024Diffusion Soup: Model Merging for Text-to-Image Diffusion Modelscited by 26
  19. 2024MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2026Group Editing: Edit Multiple Images in One Gocited by 11
  2. 2025RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scalingcited by 10
  3. 2026NaRA: Noise-Aware LoRA for Parameter-Efficient Fine-Tuning of Diffusion LLMscited by 4
  4. 2026A Unified Density Operator View of Flow Control and Mergingcited by 2
  5. 2026DuET: Dual Expert Trajectories for Diffusion Image Editingcited by 1
  6. 2026Structure-Adaptive Sparse Diffusion in Voxel Space for 3D Medical Image Enhancementcited by 0
  7. 2026MUSE: Unlocking Timestep as Native Task Steering for One-Step Dense Predictioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Adaptive Training Meets Progressive Scaling: Elevating Efficiency in Diffusion Models
  2. 2024Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoising
  3. 2023Improving Denoising Diffusion Probabilistic Models via Exploiting Shared Representations
  4. 2023Fast Inference in Denoising Diffusion Models via MMD Finetuning
  5. 2026TMPDiff: Temporal Mixed-Precision for Diffusion Models
  6. 2025Fewer Denoising Steps or Cheaper Per-Step Inference: Towards Compute-Optimal Diffusion Model Deployment
  7. 2023Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models
  8. 2024Learnable Sparsity for Vision Generative Models