# Beyond In-Place Corruption: Insertion and Deletion In Denoising Probabilistic Models

arXiv: https://arxiv.org/abs/2107.07675 · HTML: https://mlsanity.com/paper/2107.07675/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 16 Jul 2021.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 18 of 18)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019)
- [Generating Sentences by Editing Prototypes](https://mlsanity.com/paper/1709.08878/) (2017)
- [Imputer: Sequence Modelling via Imputation and Dynamic Programming](https://mlsanity.com/paper/2002.08926/) (2020)
- [EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing](https://mlsanity.com/paper/1906.08104/) (2019)
- [Learning to Represent Edits](https://mlsanity.com/paper/1810.13337/) (2018)
- [Discrete Object Generation with Reversible Inductive Construction](https://mlsanity.com/paper/1907.08268/) (2019)
- [Learning Structural Edits via Incremental Tree Transformations](https://mlsanity.com/paper/2101.12087/) (2021)
- [Optimal Completion Distillation for Sequence Learning](https://mlsanity.com/paper/1810.01398/) (2018)
- [Insertion-Deletion Transformer](https://mlsanity.com/paper/2001.05540/) (2020)
- [Task Loss Estimation for Sequence Prediction](https://mlsanity.com/paper/1511.06456/) (2015)
- [Neural Networks for Modeling Source Code Edits](https://mlsanity.com/paper/1904.02818/) (2019)

## Built on by (most cited first, 23 of 23)

- [DiGress: Discrete Denoising diffusion for graph generation](https://mlsanity.com/paper/2209.14734/) (2022)
- [A Continuous Time Framework for Discrete Denoising Models](https://mlsanity.com/paper/2205.14987/) (2022)
- [DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimization](https://mlsanity.com/paper/2302.08224/) (2023)
- [Autoregressive Diffusion Models](https://mlsanity.com/paper/2110.02037/) (2021)
- [Score-based Continuous-time Discrete Diffusion Models](https://mlsanity.com/paper/2211.16750/) (2022)
- [Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration](https://mlsanity.com/paper/2303.11435/) (2023)
- [A Reparameterized Discrete Diffusion Model for Text Generation](https://mlsanity.com/paper/2302.05737/) (2023)
- [Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning](https://mlsanity.com/paper/2410.14157/) (2024)
- [Soft Diffusion: Score Matching for General Corruptions](https://mlsanity.com/paper/2209.05442/) (2022)
- [RetroBridge: Modeling Retrosynthesis with Markov Bridges](https://mlsanity.com/paper/2308.16212/) (2023)
- [SwinGNN: Rethinking Permutation Invariance in Diffusion Models for Graph Generation](https://mlsanity.com/paper/2307.01646/) (2023)
- [DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas](https://mlsanity.com/paper/2602.01326/) (2026)
- [Classifier-free graph diffusion for molecular property targeting](https://mlsanity.com/paper/2312.17397/) (2023)
- [Formulating Discrete Probability Flow Through Optimal Transport](https://mlsanity.com/paper/2311.03886/) (2023)
- [A Diffusion Model to Shrink Proteins While Maintaining Their Function](https://mlsanity.com/paper/2511.07390/) (2025)
- [Graph Diffusion that can Insert and Delete](https://mlsanity.com/paper/2506.15725/) (2025)
- [Generative Marginalization Models](https://mlsanity.com/paper/2310.12920/) (2023)
- [Latent Diffusion Models for Controllable RNA Sequence Generation](https://mlsanity.com/paper/2409.09828/) (2024)
- [Foundations of Diffusion Models in General State Spaces: A Self-Contained Introduction](https://mlsanity.com/paper/2512.05092/) (2025)
- [Generative Molecular Morphing for Flexible-Size Design via Unbalanced Optimal Transport](https://mlsanity.com/paper/2606.07239/) (2026)
- [Synthetic location trajectory generation using categorical diffusion models](https://mlsanity.com/paper/2402.12242/) (2024)
- [ProGress: Structured Music Generation via Graph Diffusion and Hierarchical Music Analysis](https://mlsanity.com/paper/2510.10249/) (2025)
- [A Unification of Discrete, Gaussian, and Simplicial Diffusion](https://mlsanity.com/paper/2512.15923/) (2025)

## Abstract (arXiv)

Denoising diffusion probabilistic models (DDPMs) have shown impressive results on sequence generation by iteratively corrupting each example and then learning to map corrupted versions back to the original. However, previous work has largely focused on in-place corruption, adding noise to each pixel or token individually while keeping their locations the same. In this work, we consider a broader class of corruption processes and denoising models over sequence data that can insert and delete elements, while still being efficient to train and sample from. We demonstrate that these models outperform standard in-place models on an arithmetic sequence task, and that when trained on the text8 dataset they can be used to fix spelling errors without any fine-tuning.