mlsanityData to 7 Oct 2026

E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models

arXiv 2609.37533 ↗First posted 29 Sep 2026cs.CLcited by 0

On the map: Diffusion models → Discrete and text diffusion

also in 1 more
Abstract (arXiv)

Masked diffusion models (MDMs) generate sequences by progressively unmasking several tokens per denoising step, but their reverse process is typically factorized over positions, limiting sample quality in the few-step regime where diffusion's speed advantage over autoregressive decoding matters most. A recent line of work introduces a continuous Gaussian latent, trained as a variational autoencoder, to capture correlations across positions, but such approaches are prone to posterior collapse, where the latent is silently ignored. We propose Enhanced Mixture-of-Experts (E-MoE), which builds the reverse process as a mixture of factorized distributions over a discrete shared latent given by the expert-routing decisions of a Mixture-of-Experts (MoE) backbone, without increasing active parameters over the factorized baseline. Across synthetic multi-modal benchmarks, binarized MNIST, and LM1B, E-MoE improves few-step generation over factorized baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 27 of its 31 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

27 of 31 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026TUBE: Tangent Upper Bound on Evidence for Discrete Diffusion Language Models22 May
  2. 2026Breaking the Factorization Barrier in Diffusion Language Models9 Feb
  3. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner3 Oct
  4. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Model29 Sep
  5. 2025The Diffusion Duality12 Jun
  6. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  7. 2025Large Language Diffusion Models14 Feb
  8. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time28 Oct
  9. 2024Discrete Copula Diffusion2 Oct
  10. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  11. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  12. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
1 more
  1. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  7. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  8. 2021Variational Diffusion Modelscited by 1,145
  9. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  10. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  11. 2025Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modelingcited by 14
  12. 2025ReDi: Rectified Discrete Flowcited by 13
2 more
  1. 2026Self-conditioned Flow Map Language Models via Fixed-point Flowscited by 9
  2. 2026Forward-Learned Discrete Diffusion: Learning how to noise to denoise fastercited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Training of Diffusion Mixture-of-Experts Models: A Practical Recipe
  2. 2026TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Acceleration
  3. 2026LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
  4. 2026ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models
  5. 2026REFLEX: Rethinking MoE Inference as Refinement-Aware Compute Allocation in Diffusion Language Models
  6. 2026Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matching
  7. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
  8. 2026Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models