# InsNet: An Efficient, Flexible, and Performant Insertion-based Text Generation Model

arXiv: https://arxiv.org/abs/2102.11008 · HTML: https://mlsanity.com/paper/2102.11008/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 46 references that are in the map.
- First posted on arXiv 12 Feb 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019)
- [Insertion Transformer: Flexible Sequence Generation via Insertion Operations](https://mlsanity.com/paper/1902.03249/) (2019)
- [Adversarial Learning for Neural Dialogue Generation](https://mlsanity.com/paper/1701.06547/) (2017)
- [Content Planning for Neural Story Generation with Aristotelian Rescoring](https://mlsanity.com/paper/2009.09870/) (2020)
- [Non-Monotonic Sequential Text Generation](https://mlsanity.com/paper/1902.02192/) (2019)
- [End-to-End Task-Completion Neural Dialogue Systems](https://mlsanity.com/paper/1703.01008/) (2017)
- [Insertion-based Decoding with automatically Inferred Generation Order](https://mlsanity.com/paper/1902.01370/) (2019)
- [Pun Generation with Surprise](https://mlsanity.com/paper/1904.06828/) (2019)
- [Progressive Generation of Long Text with Pretrained Language Models](https://mlsanity.com/paper/2006.15720/) (2020)
- [POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training](https://mlsanity.com/paper/2005.00558/) (2020)
- [XL-Editor: Post-editing Sentences with XLNet](https://mlsanity.com/paper/1910.10479/) (2019)

## Built on by (most cited first, 14 of 14)

- [Amortizing intractable inference in large language models](https://mlsanity.com/paper/2310.04363/) (2023)
- [Tractable Control for Autoregressive Language Generation](https://mlsanity.com/paper/2304.07438/) (2023)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [Controllable Text Generation with Neurally-Decomposed Oracle](https://mlsanity.com/paper/2205.14219/) (2022)
- [A Comprehensive Survey on Data Augmentation](https://mlsanity.com/paper/2405.09591/) (2024)
- [A Pseudo-Semantic Loss for Autoregressive Models with Logical Constraints](https://mlsanity.com/paper/2312.03905/) (2023)
- [Next Steps for Human-Centered Generative AI: A Technical Perspective](https://mlsanity.com/paper/2306.15774/) (2023)
- [Flexible-length Text Infilling for Discrete Diffusion Models](https://mlsanity.com/paper/2506.13579/) (2025)
- [AMOM: Adaptive Masking over Masking for Conditional Masked Language Model](https://mlsanity.com/paper/2303.07457/) (2023)
- [Control Large Language Models via Divide and Conquer](https://mlsanity.com/paper/2410.04628/) (2024)
- [Towards More Efficient Insertion Transformer with Fractional Positional Encoding](https://mlsanity.com/paper/2112.06295/) (2021)
- [Non-autoregressive Personalized Bundle Generation](https://mlsanity.com/paper/2406.06925/) (2024)
- [Leveraging Data Augmentation and Siamese Learning for Predictive Process Monitoring](https://mlsanity.com/paper/2507.18293/) (2025)
- [Line-Coupled Language Model](https://mlsanity.com/paper/2609.07129/) (2026)

## Abstract (arXiv)

We propose InsNet, an expressive insertion-based text generator with efficient training and flexible decoding (parallel or sequential). Unlike most existing insertion-based text generation works that require re-encoding of the context after each insertion operation and thus are inefficient to train, InsNet only requires one pass of context encoding for the entire sequence during training by introducing a novel insertion-oriented position encoding and a light-weighted slot representation strategy to enable computation sharing. Furthermore, we propose an algorithm InsNet-Dinic to better determine the parallelization of insertion operations that provides a controllable switch between parallel and sequential decoding, making it flexible to handle more parallelizable tasks such as machine translation with efficient decoding, or less parallelizable tasks such as open-domain text generation to guarantee high-quality outputs. Experiments on two lexically constrained text generation datasets and three machine translation datasets demonstrate InsNet's advantages over previous insertion-based methods in terms of training speed, inference efficiency, and generation quality.