# DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformers

arXiv: https://arxiv.org/abs/2503.14487 · HTML: https://mlsanity.com/paper/2503.14487/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 31)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Building Normalizing Flows with Stochastic Interpolants](https://mlsanity.com/paper/2209.15571/) (2022)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [All are Worth Words: A ViT Backbone for Diffusion Models](https://mlsanity.com/paper/2209.12152/) (2022)
- [StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets](https://mlsanity.com/paper/2202.00273/) (2022)
- [InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation](https://mlsanity.com/paper/2309.06380/) (2023)
- [OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2402.01739/) (2024)
- [ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts](https://mlsanity.com/paper/2210.15257/) (2022)
- [Scaling Diffusion Transformers to 16 Billion Parameters](https://mlsanity.com/paper/2407.11633/) (2024)
- [Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models](https://mlsanity.com/paper/2501.11873/) (2025)
- [Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget](https://mlsanity.com/paper/2407.15811/) (2024)
- [EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing](https://mlsanity.com/paper/2410.02098/) (2024)

## Built on by (most cited first, 24 of 24)

- [Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance](https://mlsanity.com/paper/2510.24711/) (2025)
- [Simulating the Visual World with Artificial Intelligence: A Roadmap](https://mlsanity.com/paper/2511.08585/) (2025)
- [Elastic Diffusion Transformer](https://mlsanity.com/paper/2602.13993/) (2026)
- [Efficient Training of Diffusion Mixture-of-Experts Models: A Practical Recipe](https://mlsanity.com/paper/2512.01252/) (2025)
- [Route Experts by Sequence, not by Token](https://mlsanity.com/paper/2511.06494/) (2025)
- [DiverseDiT: Towards Diverse Representation Learning in Diffusion Transformers](https://mlsanity.com/paper/2603.04239/) (2026)
- [Blockwise Flow Matching: Improving Flow Matching Models For Efficient High-Quality Generation](https://mlsanity.com/paper/2510.21167/) (2025)
- [DC-DiT: Adaptive Compute and Elastic Inference for Visual Generation via Dynamic Chunking](https://mlsanity.com/paper/2603.06351/) (2026)
- [Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Models](https://mlsanity.com/paper/2604.01622/) (2026)
- [CoInteract: Physically-Consistent Human-Object Interaction Video Synthesis via Spatially-Structured Co-Generation](https://mlsanity.com/paper/2604.19636/) (2026)
- [Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoE](https://mlsanity.com/paper/2605.02641/) (2026)
- [LaTtE-Flow: Layerwise Timestep-Expert Flow-based Transformer](https://mlsanity.com/paper/2506.06952/) (2025)
- [InterMoE: Individual-Specific 3D Human Interaction Generation via Dynamic Temporal-Selective MoE](https://mlsanity.com/paper/2511.13488/) (2025)
- [Expert-Data Alignment Governs Generation Quality in Decentralized Diffusion Models](https://mlsanity.com/paper/2602.02685/) (2026)
- [DualEraser: Joint Video Object and Effect Removal via Balanced Text-Mask Guidance and Decoupled Locator-Preserver](https://mlsanity.com/paper/2605.30045/) (2026)
- [Focusing on What Matters: Saliency-Harnessing Accurate Routing for Diffusion MoE](https://mlsanity.com/paper/2606.26938/) (2026)
- [Elucidating the Role of Feature Normalization in IJEPA](https://mlsanity.com/paper/2508.02829/) (2025)
- [Dynamic Differential Linear Attention: Enhancing Linear Diffusion Transformer for High-Quality Image Generation](https://mlsanity.com/paper/2601.13683/) (2026)
- [MoECa: Aligning Feature Reuse with Expert Decomposition in Diffusion Transformers](https://mlsanity.com/paper/2606.15615/) (2026)
- [MoECodec: Image Compression for joint human and machine perception via Mixture-of-Experts](https://mlsanity.com/paper/2606.21033/) (2026)
- [MMOE: Modernizing Diffusion Transformers with Efficient Expert Design](https://mlsanity.com/paper/2607.24665/) (2026)
- [Emergent Specialization in Populations of Self-Supervised Collaborative Vision Experts Without a Shared Gate or Cross-Agent Gradients](https://mlsanity.com/paper/2609.36770/) (2026)
- [Breaking the Uniformity Trap: Scaling Video Diffusion Model via SplitMoE](https://mlsanity.com/paper/2609.38140/) (2026)
- [PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts](https://mlsanity.com/paper/2610.18857/) (2026)

## Abstract (arXiv)

Diffusion models have demonstrated remarkable success in various image generation tasks, but their performance is often limited by the uniform processing of inputs across varying conditions and noise levels. To address this limitation, we propose a novel approach that leverages the inherent heterogeneity of the diffusion process. Our method, DiffMoE, introduces a batch-level global token pool that enables experts to access global token distributions during training, promoting specialized expert behavior. To unleash the full potential of the diffusion process, DiffMoE incorporates a capacity predictor that dynamically allocates computational resources based on noise levels and sample complexity. Through comprehensive evaluation, DiffMoE achieves state-of-the-art performance among diffusion models on ImageNet benchmark, substantially outperforming both dense architectures with 3x activated parameters and existing MoE approaches while maintaining 1x activated parameters. The effectiveness of our approach extends beyond class-conditional generation to more challenging tasks such as text-to-image generation, demonstrating its broad applicability across different diffusion model applications. Project Page: https://shiml20.github.io/DiffMoE/