mlsanityData to 7 Oct 2026

Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matching

arXiv 2604.15009 ↗First posted 16 Apr 2026cs.AI cs.LGcited by 0

On the map: Diffusion models → Discrete and text diffusion

also in 1 more
Abstract (arXiv)

Flow matching retains the generation quality of diffusion models while enabling substantially faster inference, making it a compelling paradigm for generative modeling. However, when applied to language modeling, it exhibits fundamental limitations in representing complex latent distributions with irregular geometries, such as anisotropy and multimodality. To address these challenges, we propose a mixture-of-experts flow matching (MoE-FM) framework, which captures complex global transport geometries in latent space by decomposing them into locally specialized vector fields. Building on MoE-FM, we develop a non-autoregressive (NAR) language modeling approach, named YAN, instantiated with both Transformer and Mamba architectures. Across multiple downstream tasks, YAN achieves generation quality on par with both autoregressive (AR) and diffusion-based NAR language models, while requiring as few as three sampling steps. This yields a 40× speedup over AR baselines and up to a 103× speedup over diffusion language models, demonstrating substantial efficiency advantages for language modeling.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 67 of its 83 references that are in the map. First posted on arXiv 16 Apr 2026.

Builds on

67 of 83 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling1 Oct
  2. 2025Dream 7B: Diffusion Large Language Models21 Aug
  3. 2025Large Language Diffusion Models14 Feb
  4. 2025Theoretical Benefit and Limitation of Diffusion Language Model13 Feb
  5. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  6. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  7. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  8. 2023Likelihood-Based Diffusion Language Models30 May
  9. 2022SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers20 Dec
  10. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  11. 2022Flow Matching for Generative Modelingcited by 5,426
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
38 more
  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2021Convex optimizationcited by 2,479
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  9. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  10. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  11. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  12. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  13. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  14. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  15. 2017Non-Autoregressive Neural Machine Translationcited by 457
  16. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  17. 2017Wasserstein Auto-Encoderscited by 371
  18. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  19. 2024Discrete Flow Matchingcited by 301
  20. 2024Flow Matching Guide and Codecited by 291
  21. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  22. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  23. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  24. 2015Training generative neural networks via Maximum Mean Discrepancy optimizationcited by 149
  25. 2018Fast Decoding in Sequence Models using Discrete Latent Variablescited by 103
  26. 2024Consistency Flow Matching: Defining Straight Flows with Velocity Consistencycited by 101
  27. 2020Fully Non-autoregressive Neural Machine Translation: Tricks of the Tradecited by 99
  28. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 94
  29. 2019Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posteriorcited by 58
  30. 2022Efficient Video Prediction via Sparsely Conditioned Flow Matchingcited by 54
  31. 2021Lower Perplexity is Not Always Human-Likecited by 52
  32. 2024Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?cited by 45
  33. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  34. 2020Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Primingcited by 41
  35. 2015Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systemscited by 34
  36. 2025Variational Rectified Flow Matchingcited by 34
  37. 2025An Introduction to Flow Matching and Diffusion Modelscited by 33
  38. 2021A Cluster-based Approach for Improving Isotropy in Contextual Embedding Spacecited by 27

50 most cited of 57 All 57 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
  2. 2021Efficient Large Scale Language Modeling with Mixtures of Experts
  3. 2025Efficient Training of Diffusion Mixture-of-Experts Models: A Practical Recipe
  4. 2026TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Acceleration
  5. 2026Posterior Refinement: Fast Language Generation via Any-Order Flow Maps
  6. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  7. 2026REFLEX: Rethinking MoE Inference as Refinement-Aware Compute Allocation in Diffusion Language Models
  8. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Models