mlsanityData to 14 Oct 2026

SAGE: Subspace Alignment for Classifier-Free Guidance in Mixture-of-Experts Diffusion Models

arXiv 2609.34525 ↗First posted 28 Sep 2026cs.CVcited by 0

On the map: Diffusion models → Diffusion foundations and sampling also fits: Flow matching, time series and science

also in 1 more
Abstract (arXiv)

Diffusion Transformers with Mixture-of-Experts (MoE) routing are a leading recipe for scaling generative models. Classifier-Free Guidance (CFG) is essential for generation quality, yet excessively high guidance scales trigger collapse. We identify a previously unreported failure mode in their combination: the two CFG branches route independently, so their realized activations occupy different subspaces. The unconditional write then leaves the conditional subspace, and CFG amplifies that residual linearly in the guidance scale. We propose SAGE, a training-time regularizer that aligns unconditional MoE activations to the conditional subspace without restricting routing diversity, at zero inference cost. Toy experiments show that SAGE dramatically suppresses extreme drift by 9.2x. When scaled to a 1B-parameter text-to-image model, SAGE significantly improves generation quality, delivering a 9.3% boost in peak DPG-Bench performance. Extensive experiments demonstrate that SAGE consistently outperforms the baseline.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 36 of its 42 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

36 of 42 references.

Closest ancestors · 21

Same direction, newest first

  1. 2025Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance28 Oct
  2. 2025Dense2MoE: Restructuring Diffusion Transformer to MoE for Efficient Text-to-Image Generation10 Oct
  3. 2025Towards Understanding the Mechanisms of Classifier-Free Guidance25 May
  4. 2025CFG-Zero*: Improved Classifier-Free Guidance for Flow Matching Models24 Mar
  5. 2025Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts20 Mar
  6. 2025Efficient Distillation of Classifier-Free Guidance using Adapters10 Mar
  7. 2024Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models3 Oct
  8. 2024EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing2 Oct
  9. 2024Scaling Diffusion Transformers to 16 Billion Parameters16 Jul
  10. 2024No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Models2 Jul
  11. 2024CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models12 Jun
  12. 2024Guiding a Diffusion Model with a Bad Version of Itself4 Jun
9 more
  1. 2024Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models11 Apr
  2. 2024Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance8 Apr
  3. 2024Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance26 Mar
  4. 2023Characteristic Guidance: Non-linear Correction for Diffusion Model at Large Guidance Scale11 Dec
  5. 2023CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling26 Oct
  6. 2022Scalable Diffusion Models with Transformers19 Dec
  7. 2022On Distillation of Guided Diffusion Models6 Oct
  8. 2022Improving Sample Quality of Diffusion Models Using Self-Attention Guidance3 Oct
  9. 2022Classifier-Free Diffusion Guidance26 Jul

Other references · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2022Flow Matching for Generative Modelingcited by 5,702
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  6. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  11. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  12. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
3 more
  1. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  2. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  3. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 142

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Bridging the Gap: Addressing Discrepancies in Diffusion Model Training for Classifier-Free Guidance
  2. 2026Revisiting Classifier-Free Guidance Methods in Latent Diffusion Models
  3. 2026Guiding a Diffusion Model by Swapping Its Tokens
  4. 2025Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Models
  5. 2025Classifier-Free Guidance: From High-Dimensional Analysis to Generalized Guidance Forms
  6. 2025Token Perturbation Guidance for Diffusion Models
  7. 2025Dynamic Classifier-Free Diffusion Guidance via Online Feedback
  8. 2023Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models