# Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts

arXiv: https://arxiv.org/abs/2503.16057 · HTML: https://mlsanity.com/paper/2503.16057/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 40 of its 43 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)
- Mixture-of-experts / [Vision mixture-of-experts](https://mlsanity.com/moe/vision-mixture-of-experts/)

## Builds on (most cited first, 30 of 40)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets](https://mlsanity.com/paper/2406.13897/) (2024)
- [ST-MoE: Designing Stable and Transferable Sparse Expert Models](https://mlsanity.com/paper/2202.08906/) (2022)
- [BASE Layers: Simplifying Training of Large, Sparse Models](https://mlsanity.com/paper/2103.16716/) (2021)
- [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://mlsanity.com/paper/2401.09603/) (2023)
- [Photorealistic Video Generation with Diffusion Models](https://mlsanity.com/paper/2312.06662/) (2023)
- [MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer](https://mlsanity.com/paper/2303.14389/) (2023)
- [Hash Layers For Large Sparse Models](https://mlsanity.com/paper/2106.04426/) (2021)
- [Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts](https://mlsanity.com/paper/2408.15664/) (2024)
- [Fast Training of Diffusion Models with Masked Transformers](https://mlsanity.com/paper/2306.09305/) (2023)
- [RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths](https://mlsanity.com/paper/2305.18295/) (2023)

## Built on by (most cited first, 14 of 14)

- [Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance](https://mlsanity.com/paper/2510.24711/) (2025)
- [Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization](https://mlsanity.com/paper/2509.26520/) (2025)
- [Efficient Training of Diffusion Mixture-of-Experts Models: A Practical Recipe](https://mlsanity.com/paper/2512.01252/) (2025)
- [Route Experts by Sequence, not by Token](https://mlsanity.com/paper/2511.06494/) (2025)
- [UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning](https://mlsanity.com/paper/2508.18756/) (2025)
- [Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoE](https://mlsanity.com/paper/2605.02641/) (2026)
- [Rethinking Sparse Mixture of Experts from a Unified Perspective](https://mlsanity.com/paper/2503.22996/) (2025)
- [Expert-Data Alignment Governs Generation Quality in Decentralized Diffusion Models](https://mlsanity.com/paper/2602.02685/) (2026)
- [Focusing on What Matters: Saliency-Harnessing Accurate Routing for Diffusion MoE](https://mlsanity.com/paper/2606.26938/) (2026)
- [InstructMoLE: Instruction-Guided Mixture of Low-rank Experts for Multi-Conditional Image Generation](https://mlsanity.com/paper/2512.21788/) (2025)
- [MMOE: Modernizing Diffusion Transformers with Efficient Expert Design](https://mlsanity.com/paper/2607.24665/) (2026)
- [SAGE: Subspace Alignment for Classifier-Free Guidance in Mixture-of-Experts Diffusion Models](https://mlsanity.com/paper/2609.34525/) (2026)
- [ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models](https://mlsanity.com/paper/2610.01296/) (2026)
- [PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts](https://mlsanity.com/paper/2610.18857/) (2026)

## Abstract (arXiv)

Diffusion models have emerged as mainstream framework in visual generation. Building upon this success, the integration of Mixture of Experts (MoE) methods has shown promise in enhancing model scalability and performance. In this paper, we introduce Race-DiT, a novel MoE model for diffusion transformers with a flexible routing strategy, Expert Race. By allowing tokens and experts to compete together and select the top candidates, the model learns to dynamically assign experts to critical tokens. Additionally, we propose per-layer regularization to address challenges in shallow layer learning, and router similarity loss to prevent mode collapse, ensuring better expert utilization. Extensive experiments on ImageNet validate the effectiveness of our approach, showcasing significant performance gains while promising scaling properties.