mlsanityData to 13 Oct 2026

Insertion Transformer: Flexible Sequence Generation via Insertion Operations

arXiv 1902.03249 ↗First posted 8 Feb 2019cs.CL cs.LG stat.MLcited by 123

Not on the map. Why?

Cited by 21 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present the Insertion Transformer, an iterative, partially autoregressive model for sequence generation based on insertion operations. Unlike typical autoregressive models which rely on a fixed, often left-to-right ordering of the output, our approach accommodates arbitrary orderings by allowing for tokens to be inserted anywhere in the sequence during decoding. This flexibility confers a number of advantages: for instance, not only can our model be trained to follow specific orderings such as left-to-right generation or a binary tree traversal, but it can also be trained to maximize entropy over all valid insertions for robustness. In addition, our model seamlessly accommodates both fully autoregressive generation (one insertion at a time) and partially autoregressive generation (simultaneous insertions at multiple locations). We validate our approach by analyzing its performance on the WMT 2014 English-German machine translation task under various settings for training and decoding. We find that the Insertion Transformer outperforms many prior non-autoregressive approaches to translation at comparable or better levels of parallelism, and successfully recovers the performance of the original Transformer while requiring only logarithmically many iterations during decoding.

Key facts. Cited by 123 papers in the map (data through 13 Oct 2026). First posted on arXiv 8 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 32; 2022: 20; 2023: 23; 2024: 15; 2025: 12; 2026: 21.

123 papers.

Sort:

By area:

  1. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,012
  2. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 497
  3. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  4. 2022Efficient Training of Language Models to Fill in the Middlecited by 251
  5. 2022Continuous diffusion for categorical datacited by 207
  6. 2023Speculative Decoding with Big Little Decodercited by 162
  7. 2023A Reparameterized Discrete Diffusion Model for Text Generationcited by 152
  8. 2021Learning to Efficiently Sample from Diffusion Probabilistic Modelscited by 129
  9. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generationcited by 114
  10. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  11. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoisecited by 79
  12. 2021BLT: Bidirectional Layout Transformer for Controllable Layout Generationcited by 69
38 more
  1. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  2. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  3. 2025Edit Flows: Flow Matching with Edit Operationscited by 53
  4. 2021FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognitioncited by 43
  5. 2021MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAEcited by 40
  6. 2021Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervisioncited by 40
  7. 2023Learning to Reason and Memorize with Self-Notescited by 30
  8. 2021A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generationcited by 27
  9. 2021Parallel Refinements for Lexically Constrained Text Generation with BARTcited by 24
  10. 2021Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translationcited by 23
  11. 2021Learning to Complete Code with Sketchescited by 23
  12. 2021Sketch and Refine: Towards Faithful and Informative Table-to-Text Generationcited by 21
  13. 2023Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generationcited by 21
  14. 2022An Imitation Learning Curriculum for Text Editing with Non-Autoregressive Modelscited by 20
  15. 2022Why is constrained neural language generation particularly challenging?cited by 19
  16. 2024Arrows of Time for Large Language Modelscited by 19
  17. 2026DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvascited by 18
  18. 2021Thinking Clearly, Talking Fast: Concept-Guided Non-Autoregressive Generation for Open-Domain Dialogue Systemscited by 17
  19. 2024Non-autoregressive Generative Models for Reranking Recommendationcited by 17
  20. 2025Sculptor: Empowering LLMs with Cognitive Agency via Active Context Managementcited by 17
  21. 2025OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flowscited by 17
  22. 2021Learning Structural Edits via Incremental Tree Transformationscited by 16
  23. 2021Discovering Non-monotonic Autoregressive Orderings with Variational Inferencecited by 15
  24. 2022STable: Table Generation Framework for Encoder-Decoder Modelscited by 15
  25. 2021InsNet: An Efficient, Flexible, and Performant Insertion-based Text Generation Modelcited by 14
  26. 2024Speculative Decoding with CTC-based Draft Model for LLM Inference Accelerationcited by 14
  27. 2021Enriching Non-Autoregressive Transformer with Syntactic and SemanticStructures for Neural Machine Translationcited by 13
  28. 2023Candidate Soups: Fusing Candidate Results Improves Translation Quality for Non-Autoregressive Translationcited by 13
  29. 2023FiLM: Fill-in Language Models for Any-Order Generationcited by 13
  30. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inferencecited by 13
  31. 2025Sculpting Features from Noise: Reward-Guided Hierarchical Diffusion for Task-Optimal Feature Transformationcited by 13
  32. 2021IGA : An Intent-Guided Authoring Assistantcited by 12
  33. 2025Insertion Language Models: Sequence Generation with Arbitrary-Position Insertionscited by 11
  34. 2025Flexible-length Text Infilling for Discrete Diffusion Modelscited by 11
  35. 2021Exploring Dynamic Selection of Branch Expansion Orders for Code Generationcited by 10
  36. 2022ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generationcited by 10
  37. 2026CORE: Context-Robust Remasking for Diffusion Language Modelscited by 10
  38. 2021Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoringcited by 9

50 most cited of 123 All 123 (JSON)