mlsanityData to 14 Oct 2026

Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts

arXiv 2503.16057 ↗First posted 20 Mar 2025cs.CV cs.AI cs.LGcited by 14

On the map: Diffusion models → Diffusion foundations and sampling

also in 1 more
Abstract (arXiv)

Diffusion models have emerged as mainstream framework in visual generation. Building upon this success, the integration of Mixture of Experts (MoE) methods has shown promise in enhancing model scalability and performance. In this paper, we introduce Race-DiT, a novel MoE model for diffusion transformers with a flexible routing strategy, Expert Race. By allowing tokens and experts to compete together and select the top candidates, the model learns to dynamically assign experts to critical tokens. Additionally, we propose per-layer regularization to address challenges in shallow layer learning, and router similarity loss to prevent mode collapse, ensuring better expert utilization. Extensive experiments on ImageNet validate the effectiveness of our approach, showcasing significant performance gains while promising scaling properties.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 40 of its 43 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

40 of 43 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Dynamic Diffusion Transformer4 Oct
  2. 2024EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing2 Oct
  3. 2024Scaling Diffusion Transformers to 16 Billion Parameters16 Jul
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  5. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers16 Jan
  6. 2023Denoising Task Routing for Diffusion Models11 Oct
  7. 2023Fast Training of Diffusion Models with Masked Transformers15 Jun
  8. 2023Multi-Architecture Multi-Expert Diffusion Models8 Jun
  9. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer25 Mar
  10. 2022Scalable Diffusion Models with Transformers19 Dec
  11. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  12. 2021Diffusion Models Beat GANs on Image Synthesis11 May
1 more
  1. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  8. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  9. 2024Mixtral of Expertscited by 1,823
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  11. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
15 more
  1. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  2. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  3. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 455
  4. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  5. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  6. 2023Rethinking FID: Towards a Better Evaluation Metric for Image Generationcited by 310
  7. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  8. 2021Hash Layers For Large Sparse Modelscited by 237
  9. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  10. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 142
  11. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
  12. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 112
  13. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 56
  14. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 54
  15. 2024Mixture of Nested Experts: Adaptive Processing of Visual Tokenscited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 7.

14 papers.

Sort:

By area:

  1. 2025Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidancecited by 26
  2. 2025Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilizationcited by 7
  3. 2025Efficient Training of Diffusion Mixture-of-Experts Models: A Practical Recipecited by 7
  4. 2025Route Experts by Sequence, not by Tokencited by 6
  5. 2025UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learningcited by 5
  6. 2026Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoEcited by 3
  7. 2025Rethinking Sparse Mixture of Experts from a Unified Perspectivecited by 1
  8. 2026Expert-Data Alignment Governs Generation Quality in Decentralized Diffusion Modelscited by 1
  9. 2026Focusing on What Matters: Saliency-Harnessing Accurate Routing for Diffusion MoEcited by 1
  10. 2025InstructMoLE: Instruction-Guided Mixture of Low-rank Experts for Multi-Conditional Image Generationcited by 0
  11. 2026MMOE: Modernizing Diffusion Transformers with Efficient Expert Designcited by 0
  12. 2026SAGE: Subspace Alignment for Classifier-Free Guidance in Mixture-of-Experts Diffusion Modelscited by 0
2 more
  1. 2026ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Modelscited by 0
  2. 2026PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparse Mixture-of-Experts Routing in Visual Diffusion Transformers:Diagnosis, Boundary Calibration and Evolutionary Roadmap from Routing Collapse to Selective Deadlock
  2. 2025Speedrunning ImageNet Diffusion
  3. 2026DiffSparse: Accelerating Diffusion Transformers with Learned Token Sparsity
  4. 2026MoECa: Aligning Feature Reuse with Expert Decomposition in Diffusion Transformers
  5. 2025ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation
  6. 2026Rethinking Cross-Layer Information Routing in Diffusion Transformers
  7. 2024EDT: An Efficient Diffusion Transformer Framework Inspired by Human-like Sketching
  8. 2026Post-Training Pruning for Diffusion Transformers