mlsanityData to 14 Oct 2026

Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data

arXiv 2406.03736 ↗First posted 6 Jun 2024cs.LG cs.CLcited by 381

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Discrete diffusion models with absorbing processes have shown promise in language modeling. The key quantities to be estimated are the ratios between the marginal probabilities of two transitive states at all timesteps, called the concrete score. In this paper, we reveal that the concrete score in absorbing diffusion can be expressed as conditional probabilities of clean data, multiplied by a time-dependent scalar in an analytic form. Motivated by this finding, we propose reparameterized absorbing discrete diffusion (RADD), a dedicated diffusion model without time-condition that characterizes the time-independent conditional probabilities. Besides its simplicity, RADD can reduce the number of function evaluations (NFEs) by caching the output of the time-independent network when the noisy sample remains unchanged in a sampling interval, which enables sampling acceleration. Built upon the new perspective of conditional distributions, we further unify absorbing discrete diffusion and any-order autoregressive models (AO-ARMs), showing that the upper bound on the negative log-likelihood for the diffusion model can be interpreted as an expected negative log-likelihood for AO-ARMs. Further, our RADD models achieve SOTA performance among diffusion models on 5 zero-shot language modeling benchmarks (measured by perplexity) at the GPT-2 scale. Our code is available at https://github.com/ML-GSAI/RADD.

Key facts. Cited by 381 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 38 of its 53 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

38 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Fast Sampling via Discrete Non-Markov Diffusion Models with Predetermined Transition Time14 Dec
  2. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  3. 2023Likelihood-Based Diffusion Language Models30 May
  4. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  5. 2022Self-conditioned Embedding Diffusion for Text Generation8 Nov
  6. 2021Autoregressive Diffusion Models5 Oct

Other references · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2022Scalable Diffusion Models with Transformerscited by 6,054
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2016Pointer Sentinel Mixture Modelscited by 3,096
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
20 more
  1. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  2. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  3. 2023PaLM 2 Technical Reportcited by 1,162
  4. 2021Variational Diffusion Modelscited by 1,153
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  6. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  7. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  8. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 532
  9. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 502
  10. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  11. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  12. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 373
  13. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 334
  14. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  15. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 187
  16. 2022Score-based Continuous-time Discrete Diffusion Modelscited by 169
  17. 2024Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Modelscited by 160
  18. 2013A Deep and Tractable Density Estimatorcited by 68
  19. 2022Training and Inference on Any-Order Autoregressive Models the Right Waycited by 64
  20. 2024Unifying Bayesian Flow Networks and Diffusion Models through Stochastic Differential Equationscited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 158; 2026: 210.

381 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 912
  2. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  3. 2025Dream 7B: Diffusion Large Language Modelscited by 544
  4. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  5. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  6. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  7. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 274
  8. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 214
  9. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 208
  10. 2024Scaling up Masked Diffusion Models on Textcited by 199
  11. 2025Remasking Discrete Diffusion Models with Inference-Time Scalingcited by 189
  12. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
38 more
  1. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
  2. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 178
  3. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 169
  4. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 155
  5. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  6. 2025The Diffusion Dualitycited by 135
  7. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 131
  8. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 112
  9. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 88
  10. 2025A Survey on Latent Reasoningcited by 81
  11. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  12. 2025Path Planning for Masked Diffusion Model Samplingcited by 80
  13. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Timecited by 78
  14. 2025A Survey on Diffusion Language Modelscited by 78
  15. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 71
  16. 2025Generalized Interpolating Discrete Diffusioncited by 70
  17. 2025Diffusion Language Models Know the Answer Before Decodingcited by 70
  18. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 65
  19. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  20. 2024Informed Correctors for Discrete Diffusion Modelscited by 57
  21. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 57
  22. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 55
  23. 2025Edit Flows: Flow Matching with Edit Operationscited by 55
  24. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Modelscited by 54
  25. 2025Any-Order Flexible Length Masked Diffusioncited by 51
  26. 2024Distillation of Discrete Diffusion through Dimensional Correlationscited by 48
  27. 2025Diffusion Language Models are Super Data Learnerscited by 48
  28. 2025Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithmscited by 47
  29. 2025Theoretical Benefit and Limitation of Diffusion Language Modelcited by 47
  30. 2025KLASS: KL-Guided Fast Inference in Masked Diffusion Modelscited by 47
  31. 2025Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMscited by 46
  32. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inferencecited by 41
  33. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 40
  34. 2025Fine-Tuning Masked Diffusion for Provable Self-Correctioncited by 39
  35. 2025Fine-Tuning Discrete Diffusion Models with Policy Gradient Methodscited by 38
  36. 2024PepTune: De Novo Generation of Therapeutic Peptides with Multi-Objective-Guided Discrete Diffusioncited by 37
  37. 2025Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Modelscited by 37
  38. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 37

50 most cited of 381 All 381 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimizers for Diffusion Models: A Controlled Benchmark
  2. 2026Consistent Diffusion Language Models
  3. 2025Optimal Inference Schedules for Masked Diffusion Models
  4. 2026Discrete Diffusion Models: A Unified Framework from Tokenization to Generation
  5. 2024Unified Discrete Diffusion for Categorical Data
  6. 2025Inference-Time Scaling of Diffusion Language Models via Trajectory Refinement
  7. 2025Efficient Perplexity Bound and Ratio Matching in Discrete Diffusion Language Models
  8. 2025Diffusion Beats Autoregressive in Data-Constrained Settings