mlsanityData to 7 Oct 2026

Beyond In-Place Corruption: Insertion and Deletion In Denoising Probabilistic Models

arXiv 2107.07675 ↗First posted 16 Jul 2021cs.LGcited by 23

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Denoising diffusion probabilistic models (DDPMs) have shown impressive results on sequence generation by iteratively corrupting each example and then learning to map corrupted versions back to the original. However, previous work has largely focused on in-place corruption, adding noise to each pixel or token individually while keeping their locations the same. In this work, we consider a broader class of corruption processes and denoising models over sequence data that can insert and delete elements, while still being efficient to train and sample from. We demonstrate that these models outperform standard in-place models on an arithmetic sequence task, and that when trained on the text8 dataset they can be used to fix spelling errors without any fine-tuning.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 18 of its 25 references that are in the map. First posted on arXiv 16 Jul 2021.

Builds on

18 of 25 references.

References · 18

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  6. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  7. 2019Levenshtein Transformercited by 197
  8. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  9. 2017Generating Sentences by Editing Prototypescited by 88
  10. 2020Imputer: Sequence Modelling via Imputation and Dynamic Programmingcited by 65
  11. 2019EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editingcited by 63
  12. 2018Learning to Represent Editscited by 34
6 more
  1. 2019Discrete Object Generation with Reversible Inductive Constructioncited by 16
  2. 2021Learning Structural Edits via Incremental Tree Transformationscited by 16
  3. 2018Optimal Completion Distillation for Sequence Learningcited by 14
  4. 2020Insertion-Deletion Transformercited by 9
  5. 2015Task Loss Estimation for Sequence Predictioncited by 3
  6. 2019Neural Networks for Modeling Source Code Editscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 8; 2024: 3; 2025: 5; 2026: 2.

23 papers.

Sort:

By area:

  1. 2022DiGress: Discrete Denoising diffusion for graph generationcited by 503
  2. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  3. 2023DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimizationcited by 216
  4. 2021Autoregressive Diffusion Modelscited by 208
  5. 2022Score-based Continuous-time Discrete Diffusion Modelscited by 167
  6. 2023Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restorationcited by 164
  7. 2023A Reparameterized Discrete Diffusion Model for Text Generationcited by 152
  8. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  9. 2022Soft Diffusion: Score Matching for General Corruptionscited by 101
  10. 2023RetroBridge: Modeling Retrosynthesis with Markov Bridgescited by 35
  11. 2023SwinGNN: Rethinking Permutation Invariance in Diffusion Models for Graph Generationcited by 26
  12. 2026DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvascited by 16
11 more
  1. 2023Classifier-free graph diffusion for molecular property targetingcited by 13
  2. 2023Formulating Discrete Probability Flow Through Optimal Transportcited by 12
  3. 2025A Diffusion Model to Shrink Proteins While Maintaining Their Functioncited by 6
  4. 2023Generative Marginalization Modelscited by 4
  5. 2024Latent Diffusion Models for Controllable RNA Sequence Generationcited by 4
  6. 2025Graph Diffusion that can Insert and Deletecited by 4
  7. 2026Generative Molecular Morphing for Flexible-Size Design via Unbalanced Optimal Transportcited by 2
  8. 2024Synthetic location trajectory generation using categorical diffusion modelscited by 1
  9. 2025ProGress: Structured Music Generation via Graph Diffusion and Hierarchical Music Analysiscited by 1
  10. 2025Foundations of Diffusion Models in General State Spaces: A Self-Contained Introductioncited by 1
  11. 2025A Unification of Discrete, Gaussian, and Simplicial Diffusioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026A Continuous-Time Markov Chain Framework for Insertion Language Models
  2. 2026Know When to Hold 'em: Correct-Token Retention in Uniform-State Diffusion Language Models
  3. 2025Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policies
  4. 2026Let It Go or Learn to Self-Correct: Continuous Diffusion for Constrained Discrete Tasks
  5. 2023On Inference Stability for Diffusion Models
  6. 2023An Edit Friendly DDPM Noise Space: Inversion and Manipulations
  7. 2024Think While You Generate: Discrete Diffusion with Planned Denoising
  8. 2026Remask, Don't Replace: Token-to-Mask Refinement in Diffusion Language Models