mlsanityData to 14 Oct 2026

LLaDA-MoE: A Sparse MoE Diffusion Language Model

arXiv 2509.24389 ↗First posted 29 Sep 2025cs.CL cs.AIcited by 57

On the map: Diffusion models → Discrete and text diffusion

also in 1 more
Abstract (arXiv)

We introduce LLaDA-MoE, a large language diffusion model with the Mixture-of-Experts (MoE) architecture, trained from scratch on approximately 20T tokens. LLaDA-MoE achieves competitive performance with significantly reduced computational overhead by maintaining a 7B-parameter capacity while activating only 1.4B parameters during inference. Our empirical evaluation reveals that LLaDA-MoE achieves state-of-the-art performance among diffusion language models with larger parameters, surpassing previous diffusion language models LLaDA, LLaDA 1.5, and Dream across multiple benchmarks. The instruct-tuned model LLaDA-MoE-7B-A1B-Instruct demonstrates capabilities comparable to Qwen2.5-3B-Instruct in knowledge understanding, code generation, mathematical reasoning, agent and alignment tasks, despite using fewer active parameters. Our results show that integrating a sparse MoE architecture into the training objective of masked diffusion language models still brings out MoE's strengths under efficient inference with few active parameters, and opens ample room for further exploration of diffusion language models. LLaDA-MoE models are available at Huggingface.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 65 of its 71 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

65 of 71 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models8 Sep
  2. 2025Dream 7B: Diffusion Large Language Models21 Aug
  3. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models7 Jul
  4. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation25 Jun
  5. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding22 May
  6. 2025dKV-Cache: The Cache for Diffusion Language Models21 May
  7. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models15 May
  8. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  9. 2025Large Language Diffusion Models14 Feb
  10. 2024Scaling up Masked Diffusion Models on Text24 Oct
  11. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  12. 2024Simple and Effective Masked Diffusion Language Models11 Jun
6 more
  1. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  2. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  3. 2023Likelihood-Based Diffusion Language Models30 May
  4. 2022Continuous diffusion for categorical data28 Nov
  5. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  6. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  12. 2021Program Synthesis with Large Language Modelscited by 3,637
35 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2023A Survey of Large Language Modelscited by 2,576
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  7. 2024Mixtral of Expertscited by 1,823
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  9. 2020GLU Variants Improve Transformercited by 1,804
  10. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  12. 2019Root Mean Square Layer Normalizationcited by 1,480
  13. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  14. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  15. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  16. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  17. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  18. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  19. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  20. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  21. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  22. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  23. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  24. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  25. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 373
  26. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  27. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  28. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  29. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
  30. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 169
  31. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  32. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 136
  33. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 84
  34. 2024KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Taskscited by 70
  35. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 48.

57 papers.

Sort:

By area:

  1. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inferencecited by 41
  2. 2025Encoder-Decoder Diffusion Language Models for Efficient Training and Inferencecited by 24
  3. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
  4. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 18
  5. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Modelscited by 17
  6. 2026Continuous Latent Diffusion Language Modelcited by 14
  7. 2025CreditDecoding: Accelerating Parallel Decoding in Diffusion Large Language Models with Trace Creditcited by 13
  8. 2026Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Modelcited by 13
  9. 2025Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedulescited by 11
  10. 2025WAM-Diff: A Masked Diffusion VLA Framework with MoE and Online Reinforcement Learning for Autonomous Drivingcited by 10
  11. 2026LLaDA-o: An Effective and Length-Adaptive Omni Diffusion Modelcited by 10
  12. 2026MMaDA-VLA: Large Diffusion Vision-Language-Action Model with Unified Multi-Modal Instruction and Generationcited by 10
38 more
  1. 2026TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Accelerationcited by 8
  2. 2026MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decodingcited by 8
  3. 2026Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decodingcited by 7
  4. 2026ρ-textttEOS: Training-free Bidirectional Variable-Length Control for Masked Diffusion LLMscited by 7
  5. 2026The Design Space of Tri-Modal Masked Diffusion Modelscited by 6
  6. 2026FLARE: Diffusion for Hybrid Language Modelcited by 6
  7. 2025Orchestrating Dual-Boundaries: An Arithmetic Intensity Inspired Acceleration Framework for Diffusion Language Modelscited by 5
  8. 2026DARE: Diffusion Large Language Models Alignment and Reinforcement Executorcited by 5
  9. 2026Training-Trajectory-Aware Token Selectioncited by 4
  10. 2026Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMscited by 4
  11. 2026Improving Variable-Length Generation in Diffusion Language Models via Length Regularizationcited by 4
  12. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zonescited by 3
  13. 2026Swordsman: Entropy-Driven Adaptive Block Partition for Efficient Diffusion Language Modelscited by 3
  14. 2026MetaState: Persistent Working Memory Enhances Reasoning in Discrete Diffusion Language Modelscited by 3
  15. 2026Locally Coherent Parallel Decoding in Diffusion Language Modelscited by 3
  16. 2026Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Modelscited by 3
  17. 2026Mosaic: Unlocking Long-Context Inference for Diffusion LLMs via Global Memory Planning and Dynamic Peak Tamingcited by 2
  18. 2026VidLaDA: Bidirectional Diffusion Large Language Models for Efficient Video Understandingcited by 2
  19. 2026Adaptive Test-Time Compute Allocation for Block Diffusion Language Models in Complex Reasoningcited by 2
  20. 2026The Thinking Pixel: Recursive Sparse Reasoning in Multimodal Diffusion Latentscited by 2
  21. 2026Sampling Data with Chains of Forward-Backward Diffusion Stepscited by 2
  22. 2026dMoE: dLLMs with Learnable Block Expertscited by 2
  23. 2026TrustLDM: Benchmarking Trustworthiness in Language Diffusion Modelscited by 2
  24. 2026Forward-Free Diffusion Language Models with BPTT-Free Looped Refinementcited by 2
  25. 2026TimeROME-DLM: Temporal Causal Tracing and Low-Rank Inference-Time Knowledge Editing for Masked Diffusion Language Modelscited by 2
  26. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardwarecited by 2
  27. 2026GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inferencecited by 2
  28. 2026PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decodingcited by 1
  29. 2026Efficient Diffusion LLMs via Temporal-Spatial Parallel Decoding and Confidence Extrapolationcited by 1
  30. 2026DiLaServe: High SLO Attainment Serving for Diffusion Language Modelscited by 1
  31. 2026LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Modelscited by 1
  32. 2026MemDLM: Memory-Enhanced DLM Trainingcited by 0
  33. 2026When Top-1 Fails: Calibrating LoRA Monitors for Masked Diffusion LMscited by 0
  34. 2026Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMscited by 0
  35. 2026Reducing Pretraining-Generation Mismatch in Diffusion Language Modelscited by 0
  36. 2026Affix Cache for Diffusion Large Language Modelscited by 0
  37. 2026Reliability Challenges in Diffusion Vision-Language Modelscited by 0
  38. 2026Optimizing Denoising Trajectories in dLLMs: A Lightweight Evolutionary Heuristic Approachcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Training of Diffusion Mixture-of-Experts Models: A Practical Recipe
  2. 2026REFLEX: Rethinking MoE Inference as Refinement-Aware Compute Allocation in Diffusion Language Models
  3. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100B
  4. 2026Improved Large Language Diffusion Models
  5. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning
  6. 2026Think First, Diffuse Fast: Improving Diffusion Language Model Reasoning via Autoregressive Plan Conditioning
  7. 2026ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models
  8. 2026Diffusion Language Models: An Experimental Analysis