# Dense2MoE: Restructuring Diffusion Transformer to MoE for Efficient Text-to-Image Generation

arXiv: https://arxiv.org/abs/2510.09094 · HTML: https://mlsanity.com/paper/2510.09094/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 7 of its 44 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)
- Mixture-of-experts / [Vision mixture-of-experts](https://mlsanity.com/moe/vision-mixture-of-experts/)

## Builds on (most cited first, 7 of 7)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment](https://mlsanity.com/paper/2403.05135/) (2024)
- [TinyFusion: Diffusion Transformers Learned Shallow](https://mlsanity.com/paper/2412.01199/) (2024)
- [Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks](https://mlsanity.com/paper/2401.02731/) (2024)
- [BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusion](https://mlsanity.com/paper/2305.15798/) (2023)

## Built on by (most cited first, 16 of 16)

- [Elastic Diffusion Transformer](https://mlsanity.com/paper/2602.13993/) (2026)
- [Pluggable Pruning with Contiguous Layer Distillation for Diffusion Transformers](https://mlsanity.com/paper/2511.16156/) (2025)
- [TAG-MoE: Task-Aware Gating for Unified Generative Mixture-of-Experts](https://mlsanity.com/paper/2601.08881/) (2026)
- [From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation](https://mlsanity.com/paper/2605.08712/) (2026)
- [Cert-LAS: Toward Certified Model Ownership Verification for Text-to-Image Diffusion Models via Layer-Adaptive Smoothing](https://mlsanity.com/paper/2605.29809/) (2026)
- [GRASP: Guided Residual Adapters with Sample-wise Partitioning](https://mlsanity.com/paper/2512.01675/) (2025)
- [FlowInOne:Unifying Multimodal Generation as Image-in, Image-out Flow Matching](https://mlsanity.com/paper/2604.06757/) (2026)
- [TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Training](https://mlsanity.com/paper/2511.03983/) (2025)
- [Expert-Data Alignment Governs Generation Quality in Decentralized Diffusion Models](https://mlsanity.com/paper/2602.02685/) (2026)
- [Unified Safe In-context Image Generation in Multimodal Diffusion Transformers via Restricting Unsafe Information Flows](https://mlsanity.com/paper/2606.06875/) (2026)
- [PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning](https://mlsanity.com/paper/2606.10369/) (2026)
- [Importance-Aware Low-Rank Distillation of Diffusion Transformers](https://mlsanity.com/paper/2609.04646/) (2026)
- [IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts](https://mlsanity.com/paper/2609.21346/) (2026)
- [SAGE: Subspace Alignment for Classifier-Free Guidance in Mixture-of-Experts Diffusion Models](https://mlsanity.com/paper/2609.34525/) (2026)
- [Breaking the Uniformity Trap: Scaling Video Diffusion Model via SplitMoE](https://mlsanity.com/paper/2609.38140/) (2026)
- [ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models](https://mlsanity.com/paper/2610.01296/) (2026)

## Abstract (arXiv)

Diffusion Transformer (DiT) has demonstrated remarkable performance in text-to-image generation; however, its large parameter size results in substantial inference overhead. Existing parameter compression methods primarily focus on pruning, but aggressive pruning often leads to severe performance degradation due to reduced model capacity. To address this limitation, we pioneer the transformation of a dense DiT into a Mixture of Experts (MoE) for structured sparsification, reducing the number of activated parameters while preserving model capacity. Specifically, we replace the Feed-Forward Networks (FFNs) in DiT Blocks with MoE layers, reducing the number of activated parameters in the FFNs by 62.5%. Furthermore, we propose the Mixture of Blocks (MoB) to selectively activate DiT blocks, thereby further enhancing sparsity. To ensure an effective dense-to-MoE conversion, we design a multi-step distillation pipeline, incorporating Taylor metric-based expert initialization, knowledge distillation with load balancing, and group feature loss for MoB optimization. We transform large diffusion transformers (e.g., FLUX.1 [dev]) into an MoE structure, reducing activated parameters by 60% while maintaining original performance and surpassing pruning-based approaches in extensive experiments. Overall, Dense2MoE establishes a new paradigm for efficient text-to-image generation.