# Addressing Negative Transfer in Diffusion Models

arXiv: https://arxiv.org/abs/2306.00354 · HTML: https://mlsanity.com/paper/2306.00354/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 66 of its 84 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 66)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)

## Built on by (most cited first, 30 of 33)

- [Test-time Alignment of Diffusion Models without Reward Over-optimization](https://mlsanity.com/paper/2501.05803/) (2025)
- [Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure](https://mlsanity.com/paper/2410.24060/) (2024)
- [Multi-Architecture Multi-Expert Diffusion Models](https://mlsanity.com/paper/2306.04990/) (2023)
- [SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting Synthesis](https://mlsanity.com/paper/2411.16443/) (2024)
- [Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Experts](https://mlsanity.com/paper/2403.09176/) (2024)
- [A Closer Look at Time Steps is Worthy of Triple Speed-Up for Diffusion Model Training](https://mlsanity.com/paper/2405.17403/) (2024)
- [Denoising Task Routing for Diffusion Models](https://mlsanity.com/paper/2310.07138/) (2023)
- [Denoising Task Difficulty-based Curriculum for Training Diffusion Models](https://mlsanity.com/paper/2403.10348/) (2024)
- [Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyond](https://mlsanity.com/paper/2501.10945/) (2025)
- [VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting Generation with Flexible Pose and Multi-View Joint Modeling](https://mlsanity.com/paper/2503.15855/) (2025)
- [Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selection](https://mlsanity.com/paper/2409.15557/) (2024)
- [Diffusion Model Patching via Mixture-of-Prompts](https://mlsanity.com/paper/2405.17825/) (2024)
- [Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generator](https://mlsanity.com/paper/2510.13454/) (2025)
- [Improving Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architectures](https://mlsanity.com/paper/2312.09181/) (2023)
- [Diffusion Models For Multi-Modal Generative Modeling](https://mlsanity.com/paper/2407.17571/) (2024)
- [TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision](https://mlsanity.com/paper/2503.07416/) (2025)
- [Decouple-Then-Merge: Finetune Diffusion Models as Multi-Task Learning](https://mlsanity.com/paper/2410.06664/) (2024)
- [GDSG: Graph Diffusion-based Solution Generator for Optimization Problems in MEC Networks](https://mlsanity.com/paper/2412.08296/) (2024)
- [Linear Combination of Saved Checkpoints Makes Consistency and Diffusion Models Better](https://mlsanity.com/paper/2404.02241/) (2024)
- [Early-Bird Diffusion: Investigating and Leveraging Timestep-Aware Early-Bird Tickets in Diffusion Models for Efficient Training](https://mlsanity.com/paper/2504.09606/) (2025)
- [Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception](https://mlsanity.com/paper/2504.11457/) (2025)
- [Understanding, Accelerating, and Improving MeanFlow Training](https://mlsanity.com/paper/2511.19065/) (2025)
- [ComboStoc: Combinatorial Stochasticity for Diffusion Generative Models](https://mlsanity.com/paper/2405.13729/) (2024)
- [Stitched Value Model for Diffusion Alignment](https://mlsanity.com/paper/2605.19804/) (2026)
- [Efficient Personalization of Quantized Diffusion Model without Backpropagation](https://mlsanity.com/paper/2503.14868/) (2025)
- [Graceful Forgetting in Generative Language Models](https://mlsanity.com/paper/2505.19715/) (2025)
- [Exploring Time Conditioning in Diffusion Generative Models from Disjoint Noisy Data Manifolds](https://mlsanity.com/paper/2604.25289/) (2026)
- [Harnessing the Power of Training-Free Techniques in Text-to-2D Generation for Text-to-3D Generation via Score Distillation Sampling](https://mlsanity.com/paper/2505.19868/) (2025)
- [DidSee: Diffusion-Based Depth Completion for Material-Agnostic Robotic Perception and Manipulation](https://mlsanity.com/paper/2506.21034/) (2025)
- [Gradient-Aligned Calibration for Post-Training Quantization of Diffusion Models](https://mlsanity.com/paper/2602.01289/) (2026)

## Abstract (arXiv)

Diffusion-based generative models have achieved remarkable success in various domains. It trains a shared model on denoising tasks that encompass different noise levels simultaneously, representing a form of multi-task learning (MTL). However, analyzing and improving diffusion models from an MTL perspective remains under-explored. In particular, MTL can sometimes lead to the well-known phenomenon of negative transfer, which results in the performance degradation of certain tasks due to conflicts between tasks. In this paper, we first aim to analyze diffusion training from an MTL standpoint, presenting two key observations: (O1) the task affinity between denoising tasks diminishes as the gap between noise levels widens, and (O2) negative transfer can arise even in diffusion training. Building upon these observations, we aim to enhance diffusion training by mitigating negative transfer. To achieve this, we propose leveraging existing MTL methods, but the presence of a huge number of denoising tasks makes this computationally expensive to calculate the necessary per-task loss or gradient. To address this challenge, we propose clustering the denoising tasks into small task clusters and applying MTL methods to them. Specifically, based on (O2), we employ interval clustering to enforce temporal proximity among denoising tasks within clusters. We show that interval clustering can be solved using dynamic programming, utilizing signal-to-noise ratio, timestep, and task affinity for clustering objectives. Through this, our approach addresses the issue of negative transfer in diffusion models by allowing for efficient computation of MTL methods. We validate the efficacy of proposed clustering and its integration with MTL methods through various experiments, demonstrating 1) improved generation quality and 2) faster training convergence of diffusion models.