# Insertion-Deletion Transformer

arXiv: https://arxiv.org/abs/2001.05540 · HTML: https://mlsanity.com/paper/2001.05540/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jan 2020.

## Built on by (most cited first, 9 of 9)

- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes](https://mlsanity.com/paper/2111.12701/) (2021)
- [Beyond In-Place Corruption: Insertion and Deletion In Denoising Probabilistic Models](https://mlsanity.com/paper/2107.07675/) (2021)
- [Discovering Non-monotonic Autoregressive Orderings with Variational Inference](https://mlsanity.com/paper/2110.15797/) (2021)
- [Insertion Language Models: Sequence Generation with Arbitrary-Position Insertions](https://mlsanity.com/paper/2505.05755/) (2025)
- [From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models](https://mlsanity.com/paper/2603.04592/) (2026)
- [Foundations of Diffusion Models in General State Spaces: A Self-Contained Introduction](https://mlsanity.com/paper/2512.05092/) (2025)
- [Generative Molecular Morphing for Flexible-Size Design via Unbalanced Optimal Transport](https://mlsanity.com/paper/2606.07239/) (2026)
- [A Continuous-Time Markov Chain Framework for Insertion Language Models](https://mlsanity.com/paper/2606.10199/) (2026)

## Abstract (arXiv)

We propose the Insertion-Deletion Transformer, a novel transformer-based neural architecture and training method for sequence generation. The model consists of two phases that are executed iteratively, 1) an insertion phase and 2) a deletion phase. The insertion phase parameterizes a distribution of insertions on the current output hypothesis, while the deletion phase parameterizes a distribution of deletions over the current output hypothesis. The training method is a principled and simple algorithm, where the deletion model obtains its signal directly on-policy from the insertion model output. We demonstrate the effectiveness of our Insertion-Deletion Transformer on synthetic translation tasks, obtaining significant BLEU score improvement over an insertion-only model.