mlsanityData to 14 Oct 2026

DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone

arXiv 2511.15927 ↗First posted 19 Nov 2025cs.LG cs.AIcited by 6

On the map: Diffusion models → Discrete and text diffusion

also in 2 more
Abstract (arXiv)

Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) generation, yet their reliance on Transformer backbones limits inference efficiency due to quadratic attention or KV-cache overhead. We introduce DiffuMamba, a masked diffusion language model built on a bidirectional Mamba backbone that combines the diffusion objective with linear-time sequence modeling, and DiffuMamba-H, a hybrid variant with interleaved attention. Across scales up to 1.3B parameters, our models match Transformer-based diffusion in downstream performance while achieving up to 8.2x and 4.3x higher inference throughput, respectively, on long sequences. We further present a systematic analysis of inference efficiency across modern DLM variants combining asymptotic complexity with empirical measurements. Notably, cache-efficient block diffusion with Mamba mixers emerges as the only strategy that scales linearly with sequence length and achieves the strongest performance across all baselines, suggesting a promising direction for future diffusion-based generation systems.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 74 of its 81 references that are in the map. First posted on arXiv 19 Nov 2025.

Builds on

74 of 81 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Diffusion Language Models are Super Data Learners5 Nov
  2. 2025Training Optimal Large Diffusion Language Models28 Sep
  3. 2025Dream 7B: Diffusion Large Language Models21 Aug
  4. 2025A Survey on Diffusion Language Models14 Aug
  5. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing8 Aug
  6. 2025The Diffusion Duality12 Jun
  7. 2025dKV-Cache: The Cache for Diffusion Language Models21 May
  8. 2025Large Language Diffusion Models14 Feb
  9. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  10. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  11. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  12. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
3 more
  1. 2024Glauber Generative Model: Discrete Diffusion Models via Binary Classification27 May
  2. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  3. 2023A Cheaper and Better Diffusion Language Model with Soft-Masked Noise10 Apr

Other references · 59

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2022Scalable Diffusion Models with Transformerscited by 6,054
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
38 more
  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  11. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  12. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  13. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  14. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  15. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  16. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  17. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  18. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  19. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  20. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  21. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  22. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  23. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  24. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  25. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  26. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  27. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  28. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 169
  29. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 113
  30. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 112
  31. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 100
  32. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  33. 2024DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesiscited by 72
  34. 2025Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performancecited by 57
  35. 2024Vision Mamba: A Comprehensive Survey and Taxonomycited by 54
  36. 2024Visual Mamba: A Survey and New Outlookscited by 41
  37. 2025A Systematic Analysis of Hybrid Linear Attentioncited by 39
  38. 2024LKM-UNet: Large Kernel Vision Mamba UNet for Medical Image Segmentationcited by 34

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026FLARE: Diffusion for Hybrid Language Modelcited by 6
  2. 2026Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scalecited by 2
  3. 2026Training Hybrid Block Diffusion Language Models with Partial Bidirectionalitycited by 1
  4. 2026Anatomy of Associative Recall in Fixed-State Recurrences: A Matched-State Decomposition, an Interference Wall, and a Curriculum That Breaks Itcited by 1
  5. 2026DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Modelscited by 0
  6. 2026RL Post-Training Diffusion Language Models Along Their Own Generation Pathscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques
  2. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMs
  3. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion
  4. 2025How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices
  5. 2025Diffusion Language Model Inference with Monte Carlo Tree Search
  6. 2025Sequential Diffusion Language Models
  7. 2026Diffusion Language Models: An Experimental Analysis
  8. 2025Diffusion Language Models are Provably Optimal Parallel Samplers