mlsanityData to 14 Oct 2026

Multi-Architecture Multi-Expert Diffusion Models

arXiv 2306.04990 ↗First posted 8 Jun 2023cs.CVcited by 35

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

In this paper, we address the performance degradation of efficient diffusion models by introducing Multi-architecturE Multi-Expert diffusion models (MEME). We identify the need for tailored operations at different time-steps in diffusion processes and leverage this insight to create compact yet high-performing models. MEME assigns distinct architectures to different time-step intervals, balancing convolution and self-attention operations based on observed frequency characteristics. We also introduce a soft interval assignment strategy for comprehensive training. Empirically, MEME operates 3.3 times faster than baselines while improving image generation quality (FID scores) by 0.62 (FFHQ) and 0.37 (CelebA). Though we validate the effectiveness of assigning more optimal architecture per time-step, where efficient models outperform the larger models, we argue that MEME opens a new design choice for diffusion models that can be easily applied in other scenarios, such as large multi-expert models.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 60 of its 74 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

60 of 74 references.

Closest ancestors · 29

Same direction, newest first

  1. 2023Addressing Negative Transfer in Diffusion Models1 Jun
  2. 2023Efficient Diffusion Training via Min-SNR Weighting Strategy16 Mar
  3. 2023Consistency Models2 Mar
  4. 2022Exploring Vision Transformers as Diffusion Learners28 Dec
  5. 2022Towards Practical Plug-and-Play Diffusion Models12 Dec
  6. 2022Post-training Quantization on Diffusion Models28 Nov
  7. 2022Diffusion Probabilistic Model Made Slim27 Nov
  8. 2022GENIE: Higher-Order Denoising Diffusion Solvers11 Oct
  9. 2022On Distillation of Guided Diffusion Models6 Oct
  10. 2022Accelerating Score-based Generative Models with Preconditioned Diffusion Sampling5 Jul
  11. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  12. 2022Improved Vector Quantized Diffusion Models31 May
17 more
  1. 2022Accelerating Diffusion Models via Early Stop of the Diffusion Process25 May
  2. 2022Perception Prioritized Training of Diffusion Models1 Apr
  3. 2022Pseudo Numerical Methods for Diffusion Models on Manifolds20 Feb
  4. 2022Truncated Diffusion Probabilistic Models and Diffusion-based Adversarial Auto-Encoders19 Feb
  5. 2022Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality11 Feb
  6. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models17 Jan
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  8. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesis29 Nov
  9. 2021Score-based Generative Modeling in Latent Space10 Jun
  10. 2021Learning to Efficiently Sample from Diffusion Probabilistic Models7 Jun
  11. 2021On Fast Sampling of Diffusion Probabilistic Models31 May
  12. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  13. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  14. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed7 Jan
  15. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  16. 2020Denoising Diffusion Implicit Models6 Oct
  17. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  10. 2014Going Deeper with Convolutionscited by 4,241
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  12. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
19 more
  1. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  2. 2022Video Diffusion Modelscited by 2,305
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  4. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,512
  5. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  6. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  7. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  8. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  9. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  10. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 553
  11. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  12. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 465
  13. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 447
  14. 2022How Do Vision Transformers Work?cited by 307
  15. 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
  16. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesiscited by 136
  17. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generationcited by 133
  18. 2022Inception Transformercited by 99
  19. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidancecited by 84

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 16; 2025: 12; 2026: 4.

35 papers.

Sort:

By area:

  1. 2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 66
  2. 2024Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learningcited by 48
  3. 2024SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting Synthesiscited by 34
  4. 2024T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitchingcited by 33
  5. 2024Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Expertscited by 31
  6. 2024EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routingcited by 30
  7. 2023Denoising Task Routing for Diffusion Modelscited by 28
  8. 2024Denoising Task Difficulty-based Curriculum for Training Diffusion Modelscited by 27
  9. 2025Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidancecited by 26
  10. 2024Beta Sampling is All You Need: Efficient Image Generation Strategy for Diffusion Models using Stepwise Spectral Analysiscited by 19
  11. 2025VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting Generation with Flexible Pose and Multi-View Joint Modelingcited by 14
  12. 2025Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Expertscited by 14
23 more
  1. 2024Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Modelscited by 13
  2. 2024Ensembling Diffusion Models via Adaptive Feature Aggregationcited by 12
  3. 2024Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selectioncited by 12
  4. 2025Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generatorcited by 11
  5. 2023Improving Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architecturescited by 9
  6. 2025TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Visioncited by 9
  7. 2024Memory-Efficient Fine-Tuning for Quantized Diffusion Modelcited by 8
  8. 2024Decouple-Then-Merge: Finetune Diffusion Models as Multi-Task Learningcited by 7
  9. 2025FeRA: Frequency-Energy Constrained Routing for Effective Diffusion Adaptation Fine-Tuningcited by 7
  10. 2023Generative AI Beyond LLMs: System Implications of Multi-Modal Generationcited by 6
  11. 2024FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Modelscited by 5
  12. 2024Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoisingcited by 5
  13. 2025Early-Bird Diffusion: Investigating and Leveraging Timestep-Aware Early-Bird Tickets in Diffusion Models for Efficient Trainingcited by 5
  14. 2024Diffusion based Semantic Outlier Generation via Nuisance Awareness for Out-of-Distribution Detectioncited by 4
  15. 2025Monte Carlo Tree Diffusion with Multiple Experts for Protein Designcited by 2
  16. 2026Stitched Value Model for Diffusion Alignmentcited by 2
  17. 2025FlexControl: Computation-Aware ControlNet with Differentiable Router for Text-to-Image Generationcited by 1
  18. 2025Efficient Personalization of Quantized Diffusion Model without Backpropagationcited by 1
  19. 2025LaTtE-Flow: Layerwise Timestep-Expert Flow-based Transformercited by 1
  20. 2025InterMoE: Individual-Specific 3D Human Interaction Generation via Dynamic Temporal-Selective MoEcited by 1
  21. 2026Complexity-Balanced Diffusion Splittingcited by 1
  22. 2026MoECodec: Image Compression for joint human and machine perception via Mixture-of-Expertscited by 0
  23. 2026PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practices
  2. 2023Analyzing and Improving the Training Dynamics of Diffusion Models
  3. 2025Paris: A Decentralized Trained Open-Weight Diffusion Model
  4. 2024Accelerating Diffusion Models with One-to-Many Knowledge Distillation
  5. 2022Diffusion Models: A Comprehensive Survey of Methods and Applications
  6. 2024On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models
  7. 2024SDXS: Real-Time One-Step Latent Diffusion Models with Image Conditions
  8. 2025Glance: Accelerating Diffusion Models with 1 Sample