mlsanityData to 7 Oct 2026

Continuous Diffusion Model for Language Modeling

arXiv 2502.11564 ↗First posted 17 Feb 2025cs.LGcited by 29

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion models have emerged as a promising alternative to autoregressive models in modeling discrete categorical data. However, diffusion models that directly work on discrete data space fail to fully exploit the power of iterative refinement, as the signals are lost during transitions between discrete states. Existing continuous diffusion models for discrete data underperform compared to discrete methods, and the lack of a clear connection between the two approaches hinders the development of effective diffusion models for discrete data. In this work, we propose a continuous diffusion model for language modeling that incorporates the geometry of the underlying categorical distribution. We establish a connection between the discrete diffusion and continuous flow on the statistical manifold, and building on this analogy, introduce a simple diffusion process that generalizes existing discrete diffusion models. We further propose a simulation-free training framework based on radial symmetry, along with a simple technique to address the high dimensionality of the manifold. Comprehensive experiments on language modeling benchmarks and other modalities show that our method outperforms existing discrete diffusion models and approaches the performance of autoregressive models. The code is available at https://github.com/harryjo97/RDLM.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 55 of its 66 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

55 of 66 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  2. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  3. 2023Likelihood-Based Diffusion Language Models30 May
  4. 2022Continuous diffusion for categorical data28 Nov
  5. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  6. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct
  7. 2021Autoregressive Diffusion Models5 Oct

Other references · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2022Scalable Diffusion Models with Transformerscited by 5,931
  6. 2022Classifier-Free Diffusion Guidancecited by 5,717
  7. 2022Flow Matching for Generative Modelingcited by 5,426
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  11. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
36 more
  1. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  3. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  4. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  5. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  6. 2016Pixel Recurrent Neural Networkscited by 814
  7. 2018Image Transformercited by 770
  8. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  9. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  10. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  11. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  12. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 413
  13. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  14. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  15. 2024Discrete Flow Matchingcited by 301
  16. 2023Flow Matching on General Geometriescited by 260
  17. 2022Riemannian Score-Based Generative Modellingcited by 216
  18. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  19. 2023On the Importance of Noise Scheduling for Diffusion Modelscited by 193
  20. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  21. 2017PixelSNAIL: An Improved Autoregressive Generative Modelcited by 137
  22. 2024Dirichlet Flow Matching with Applications to DNA Sequence Designcited by 110
  23. 2023Non-Denoising Forward-Time Diffusionscited by 75
  24. 2023Dirichlet Diffusion Score Model for Biological Sequence Generationcited by 73
  25. 2019Discrete Flows: Invertible Generative Models of Discrete Datacited by 68
  26. 2024Fisher Flow Matching for Generative Modeling over Discrete Datacited by 68
  27. 2019Latent Normalizing Flows for Discrete Sequencescited by 66
  28. 2022Training and Inference on Any-Order Autoregressive Models the Right Waycited by 64
  29. 2023Bayesian Flow Networkscited by 62
  30. 2024Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspectivecited by 54
  31. 2023Graph Generation with Diffusion Mixturecited by 52
  32. 2024OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flowscited by 45
  33. 2024Categorical Flow Matching on Statistical Manifoldscited by 43
  34. 2021OmniNet: Omnidirectional Representations from Transformerscited by 23
  35. 2023Generative Modeling on Manifolds Through Mixture of Riemannian Diffusion Processescited by 21
  36. 2024Conditional Synthesis of 3D Molecules with Time Correction Samplercited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 24.

29 papers.

Sort:

By area:

  1. 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modelingcited by 32
  2. 2026LangFlow: Continuous Diffusion Rivals Discrete in Language Modelingcited by 28
  3. 2026ELF: Embedded Language Flowscited by 27
  4. 2025CtrlDiff: Boosting Large Diffusion Language Models with Dynamic Block Prediction and Controllable Generationcited by 17
  5. 2026Continuous Diffusion Scales Competitively with Discrete Diffusion for Languagecited by 13
  6. 2026Continuous Latent Diffusion Language Modelcited by 12
  7. 2025Efficient Parallel Samplers for Recurrent-Depth Models and Their Connection to Diffusion Language Modelscited by 7
  8. 2026Spherical Flows for Sampling Categorical Datacited by 4
  9. 2026CoBit: Language Modeling with Bitstream Diffusioncited by 4
  10. 2025Masked Diffusion Models as Energy Minimizationcited by 3
  11. 2026DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modelingcited by 3
  12. 2026Do Diffusion Models Dream of Electric Planes? Discrete and Continuous Simulation-Based Inference for Aircraft Designcited by 2
17 more
  1. 2026Low Perplexity is Repetition: A One-Dimensional Self-Conditioning Attractor in Continuous Diffusion LMscited by 2
  2. 2026Simplex Relaxation for Discrete Diffusioncited by 2
  3. 2026Distilled Continuous Diffusion Language Models Can Write Code in Few Steps---or Onecited by 2
  4. 2025Foundations of Diffusion Models in General State Spaces: A Self-Contained Introductioncited by 1
  5. 2026Where Should Diffusion Enter a Language Model? Geometry-Guided Hidden-State Replacementcited by 1
  6. 2026VoidPadding: Let [VOID] Handle Padding in Masked Diffusion Language Models so that [EOS] Can Focus on Semantic Terminationcited by 1
  7. 2026Multi-Mask Diffusion Language Models for Few-Step Generationcited by 1
  8. 2026Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMscited by 1
  9. 2026Lost in Interpolation: Why Predictive Feedback Fails in Diffusion Language Modelscited by 1
  10. 2026Continuous Diffusion Models Can Obey Formal Syntaxcited by 0
  11. 2026Language Generation as Optimal Control: Closed-Loop Diffusion in Latent Control Spacecited by 0
  12. 2026The Attribution Contract for Generative Language Modelscited by 0
  13. 2026Diffusion Image Generation with Explicit Modeling of Data Manifold Geometrycited by 0
  14. 2026Dynamic Infilling Anchors for Format-Constrained Generation in Diffusion Large Language Modelscited by 0
  15. 2026Token Time Continuous Diffusion for Language Modelingcited by 0
  16. 2026Diffusion Language Model for Recommendationcited by 0
  17. 2026Temporally-Resolved Token Attribution Reveals the Generation Dynamics of Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimality of FSQ Tokens for Continuous Diffusion for Categorical Data with Application to Text-to-Speech
  2. 2025Efficient Perplexity Bound and Ratio Matching in Discrete Diffusion Language Models
  3. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models
  4. 2023A Survey of Diffusion Models in Natural Language Processing
  5. 2025Discrete Diffusion Models for Language Generation
  6. 2026Scaling Categorical Flow Maps
  7. 2026Consistent Diffusion Language Models
  8. 2026Flow Map Language Models: One-step Language Modeling via Continuous Denoising