# Generalized Interpolating Discrete Diffusion

arXiv: https://arxiv.org/abs/2503.04482 · HTML: https://mlsanity.com/paper/2503.04482/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 33 of its 52 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution](https://mlsanity.com/paper/2310.16834/) (2023)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [A Continuous Time Framework for Discrete Denoising Models](https://mlsanity.com/paper/2205.14987/) (2022)
- [Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data](https://mlsanity.com/paper/2406.03736/) (2024)
- [Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation](https://mlsanity.com/paper/2303.00848/) (2023)
- [Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions](https://mlsanity.com/paper/2502.06768/) (2025)
- [Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling](https://mlsanity.com/paper/2409.02908/) (2024)
- [Scaling up Masked Diffusion Models on Text](https://mlsanity.com/paper/2410.18514/) (2024)
- [An Actor-Critic Algorithm for Sequence Prediction](https://mlsanity.com/paper/1607.07086/) (2016)
- [Likelihood-Based Diffusion Language Models](https://mlsanity.com/paper/2305.18619/) (2023)
- [Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs](https://mlsanity.com/paper/2407.01082/) (2024)

## Built on by (most cited first, 30 of 70)

- [A Survey on Diffusion Language Models](https://mlsanity.com/paper/2508.10875/) (2025)
- [Discrete Diffusion in Large Language and Multimodal Models: A Survey](https://mlsanity.com/paper/2506.13759/) (2025)
- [Fine-Tuning Masked Diffusion for Provable Self-Correction](https://mlsanity.com/paper/2510.01384/) (2025)
- [Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Models](https://mlsanity.com/paper/2509.23653/) (2025)
- [Scaling Behavior of Discrete Diffusion Language Models](https://mlsanity.com/paper/2512.10858/) (2025)
- [Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner](https://mlsanity.com/paper/2510.03206/) (2025)
- [Cosmos: Compressed and Smooth Latent Space for Text Diffusion Modeling](https://mlsanity.com/paper/2506.21170/) (2025)
- [DMax: Aggressive Parallel Decoding for dLLMs](https://mlsanity.com/paper/2604.08302/) (2026)
- [Anchored Diffusion Language Model](https://mlsanity.com/paper/2505.18456/) (2025)
- [Soft-Masked Diffusion Language Models](https://mlsanity.com/paper/2510.17206/) (2025)
- [Continuous Latent Diffusion Language Model](https://mlsanity.com/paper/2605.06548/) (2026)
- [Learn from Your Mistakes: Self-Correcting Masked Diffusion Models](https://mlsanity.com/paper/2602.11590/) (2026)
- [Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation](https://mlsanity.com/paper/2605.22765/) (2026)
- [The Diffusion Duality, Chapter II: Psi-Samplers](https://mlsanity.com/paper/2602.21185/) (2026)
- [Corrective Diffusion Language Models](https://mlsanity.com/paper/2512.15596/) (2025)
- [CORE: Context-Robust Remasking for Diffusion Language Models](https://mlsanity.com/paper/2602.04096/) (2026)
- [Smoothie: Smoothing Diffusion on Token Embeddings for Text Generation](https://mlsanity.com/paper/2505.18853/) (2025)
- [Next Semantic Scale Prediction via Hierarchical Diffusion Language Models](https://mlsanity.com/paper/2510.08632/) (2025)
- [Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall](https://mlsanity.com/paper/2510.19304/) (2025)
- [IDLM: Inverse-distilled Diffusion Language Models](https://mlsanity.com/paper/2602.19066/) (2026)
- [Masked Diffusion for Generative Recommendation](https://mlsanity.com/paper/2511.23021/) (2025)
- [Partition Generative Modeling: Masked Modeling Without Masks](https://mlsanity.com/paper/2505.18883/) (2025)
- [The Design Space of Tri-Modal Masked Diffusion Models](https://mlsanity.com/paper/2602.21472/) (2026)
- [JointDiff: Bridging Continuous and Discrete in Multi-Agent Trajectory Generation](https://mlsanity.com/paper/2509.22522/) (2025)
- [Finish First, Perfect Later: Test-Time Token-Level Cross-Validation for Diffusion Large Language Models](https://mlsanity.com/paper/2510.05090/) (2025)
- [Guided Star-Shaped Masked Diffusion](https://mlsanity.com/paper/2510.08369/) (2025)
- [ILRR: Inference-Time Steering Method for Masked Diffusion Language Models](https://mlsanity.com/paper/2601.21647/) (2026)
- [Provably adaptive sampling with uniform and remasking discrete diffusion models](https://mlsanity.com/paper/2608.23554/) (2026)
- [Scalable Non-Equivariant 3D Molecule Generation via Rotational Alignment](https://mlsanity.com/paper/2506.10186/) (2025)
- [DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generation](https://mlsanity.com/paper/2510.12210/) (2025)

## Abstract (arXiv)

While state-of-the-art language models achieve impressive results through next-token prediction, they have inherent limitations such as the inability to revise already generated tokens. This has prompted exploration of alternative approaches such as discrete diffusion. However, masked diffusion, which has emerged as a popular choice due to its simplicity and effectiveness, reintroduces this inability to revise words. To overcome this, we generalize masked diffusion, deriving a new family of general interpolating discrete diffusion (GIDD) which offers greater flexibility in the design of the noising processes. Leveraging a novel diffusion ELBO, we achieve compute-matched state-of-the-art performance in diffusion language modeling. Exploiting GIDD's flexibility, we explore a hybrid approach combining masking and uniform noise, leading to improved sample quality and unlocking the ability for the model to correct its own mistakes, an area where autoregressive models notoriously have struggled. Code: https://github.com/dvruette/gidd/