mlsanityData to 14 Oct 2026

Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling

arXiv 2510.01329 ↗First posted 1 Oct 2025stat.ML cs.LGcited by 34

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Standard discrete diffusion models treat all unobserved states identically by mapping them to an absorbing [MASK] token. This creates an 'information void' where semantic information that could be inferred from unmasked tokens is lost between denoising steps. We introduce Continuously Augmented Discrete Diffusion (CADD), a framework that augments the discrete state space with a paired diffusion in a continuous latent space. This yields graded, gradually corrupted states in which masked tokens are represented by noisy yet informative latent vectors rather than collapsed 'information voids'. At each reverse step, CADD may leverage the continuous latent as a semantic hint to guide discrete denoising. The design is clean and compatible with existing discrete diffusion training. At sampling time, the strength and choice of estimator for the continuous latent vector enables a controlled trade-off between mode-coverage (generating diverse outputs) and mode-seeking (generating contextually precise outputs) behaviors. Empirically, we demonstrate CADD improves generative quality over mask-based diffusion across text generation, image synthesis, and code modeling, with consistent gains on both qualitative and quantitative metrics against strong discrete baselines.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 82 of its 89 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

82 of 89 references.

Closest ancestors · 24

Same direction, newest first

  1. 2025Dream 7B: Diffusion Large Language Models21 Aug
  2. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference4 Aug
  3. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation25 Jun
  4. 2025The Diffusion Duality12 Jun
  5. 2025Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking24 May
  6. 2025Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion23 Apr
  7. 2025Remasking Discrete Diffusion Models with Inference-Time Scaling1 Mar
  8. 2025TESS 2: A Large-Scale Generalist Diffusion Language Model19 Feb
  9. 2025Continuous Diffusion Model for Language Modeling17 Feb
  10. 2025Large Language Diffusion Models14 Feb
  11. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time28 Oct
  12. 2024Scaling up Masked Diffusion Models on Text24 Oct
12 more
  1. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  2. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  3. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  4. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model5 Jun
  5. 2023Likelihood-Based Diffusion Language Models30 May
  6. 2023TESS: Text-to-Text Self-Conditioned Simplex Diffusion15 May
  7. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
  8. 2022Empowering Diffusion Models on the Embedding Space for Text Generation19 Dec
  9. 2022Continuous diffusion for categorical data28 Nov
  10. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  11. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct
  12. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 58

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2022Flow Matching for Generative Modelingcited by 5,702
  10. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  11. 2021Program Synthesis with Large Language Modelscited by 3,637
  12. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
38 more
  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  2. 2023Consistency Modelscited by 1,822
  3. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  4. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  5. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  6. 2021Variational Diffusion Modelscited by 1,153
  7. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  8. 2023One-step Diffusion with Distribution Matching Distillationcited by 1,019
  9. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  10. 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
  11. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  12. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  13. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 516
  14. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  15. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  16. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 373
  17. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 363
  18. 2024Discrete Flow Matchingcited by 317
  19. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  20. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Modelscited by 236
  21. 2024Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generationcited by 214
  22. 2022Diffusion-GAN: Training GANs with Diffusioncited by 205
  23. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 130
  24. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 129
  25. 2024Dirichlet Flow Matching with Applications to DNA Sequence Designcited by 115
  26. 2021Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimationcited by 97
  27. 2024Normalizing Flows are Capable Generative Modelscited by 91
  28. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 84
  29. 2023Dirichlet Diffusion Score Model for Biological Sequence Generationcited by 75
  30. 2024Fisher Flow Matching for Generative Modeling over Discrete Datacited by 72
  31. 2022Training and Inference on Any-Order Autoregressive Models the Right Waycited by 64
  32. 2024Informed Correctors for Discrete Diffusion Modelscited by 57
  33. 2025Edit Flows: Flow Matching with Edit Operationscited by 55
  34. 2023Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEscited by 54
  35. 2024Categorical Flow Matching on Statistical Manifoldscited by 47
  36. 2023Mirror Diffusion Models for Constrained and Watermarked Generationcited by 42
  37. 2025STARFlow: Scaling Latent Normalizing Flows for High-resolution Image Synthesiscited by 32
  38. 2019Self-supervised GAN: Analysis and Improvement with Multi-class Minimax Gamecited by 25

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 27.

34 papers.

Sort:

By area:

  1. 2026ELF: Embedded Language Flowscited by 45
  2. 2025CANDI: Hybrid Discrete-Continuous Diffusion Modelscited by 34
  3. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 20
  4. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
  5. 2025Soft-Masked Diffusion Language Modelscited by 16
  6. 2026CoDAR: Continuous Diffusion Language Models are More Powerful Than You Thinkcited by 14
  7. 2026Continuous Latent Diffusion Language Modelcited by 14
  8. 2026Beyond Hard Masks: Progressive Token Evolution for Diffusion Language Modelscited by 6
  9. 2026The Design Space of Tri-Modal Masked Diffusion Modelscited by 6
  10. 2026Rejection Mixing: Fast Semantic Propagation of Mask Tokens for Efficient DLLM Inferencecited by 4
  11. 2026DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modelingcited by 4
  12. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zonescited by 3
22 more
  1. 2026MetaState: Persistent Working Memory Enhances Reasoning in Discrete Diffusion Language Modelscited by 3
  2. 2025Foundations of Diffusion Models in General State Spaces: A Self-Contained Introductioncited by 2
  3. 2026Masked Language Flow Modelscited by 2
  4. 2026Sticky Jump Diffusions: A Unifying View of Masked, Continuous, and Hybrid Diffusioncited by 2
  5. 2026Simplex Relaxation for Discrete Diffusioncited by 2
  6. 2026Distilled Continuous Diffusion Language Models Can Write Code in Few Steps---or Onecited by 2
  7. 2026Insertion Based Sequence Generation with Learnable Order Dynamicscited by 1
  8. 2026Binomial flows: Denoising and flow matching for discrete ordinal datacited by 1
  9. 2026Infinite Mask Diffusion for Few-Step Distillationcited by 1
  10. 2026Learned Relay Representations for Forward-Thinking Discrete Diffusion Modelscited by 1
  11. 2026Token Time Continuous Diffusion for Language Modelingcited by 1
  12. 2026Multi-Mask Diffusion Language Models for Few-Step Generationcited by 1
  13. 2026From Interface to Inference: Eliciting Any-Order Inference from Any-Order Modelscited by 1
  14. 2026ELF-REG: Scaling Continuous Diffusion Language Models to Reasoning Taskscited by 1
  15. 2025Latent-Augmented Discrete Diffusion Modelscited by 0
  16. 2026Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matchingcited by 0
  17. 2026Understanding and Accelerating the Training of Masked Diffusion Language Modelscited by 0
  18. 2026Diffusion Language Model for Recommendationcited by 0
  19. 2026Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Modelscited by 0
  20. 2026Hierarchical Continuous Diffusion Language Modelscited by 0
  21. 2026Jumping up and down: Denoiser diffusion models for discrete ordinal datacited by 0
  22. 2026On the Design Space of Continuous Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models
  2. 2025Discrete Diffusion Models for Language Generation
  3. 2026Bridging the Discrete-Continuous Gap: Unified Multimodal Generation via Coupled Manifold Discrete Absorbing Diffusion
  4. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling
  5. 2026Balancing Understanding and Generation in Discrete Diffusion Models
  6. 2026AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling
  7. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner
  8. 2023A Cheaper and Better Diffusion Language Model with Soft-Masked Noise