mlsanityData to 14 Oct 2026

A Continuous-Time Markov Chain Framework for Insertion Language Models

arXiv 2606.10199 ↗First posted 8 Jun 2026cs.LG cs.CLcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Insertion Language Models (ILMs) offer several advantages over left-to-right generation and mask-based generation. However, existing formulations of insertion-based generation have largely been ad-hoc. In this paper, we derive a diffusion-style denoising objective for ILMs from first principles by formulating the noising process as a continuous-time Markov chain on the space of variable-length sequences. We show that previous formulations of ILMs can be viewed as special cases of this denoising framework. Through empirical evaluation on a synthetic planning task, we show that the proposed approach retains the benefits of insertion-based generation over left-to-right generation and masked diffusion models. In language modeling, our diffusion-based approach is competitive with left-to-right generation and masked diffusion models, while offering additional flexibility in sampling compared to existing insertion language models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 31 of its 50 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

31 of 50 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas1 Feb
  2. 2026ρ-textttEOS: Training-free Bidirectional Variable-Length Control for Masked Diffusion LLMs30 Jan
  3. 2025Any-Order Flexible Length Masked Diffusion31 Aug
  4. 2024Scaling up Masked Diffusion Models on Text24 Oct
  5. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  6. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling4 Sep
  7. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  8. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  9. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  10. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  11. 2022DiffusER: Discrete Diffusion via Edit-based Reconstruction30 Oct

Other references · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  7. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  8. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 939
  9. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  10. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  11. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 305
  12. 2024The pitfalls of next-token predictioncited by 160
8 more
  1. 2019Insertion Transformer: Flexible Sequence Generation via Insertion Operationscited by 125
  2. 2019Non-Monotonic Sequential Text Generationcited by 55
  3. 2025Edit Flows: Flow Matching with Edit Operationscited by 55
  4. 2023Trans-Dimensional Generative Modeling via Jump Diffusion Modelscited by 35
  5. 2025Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Modelscited by 34
  6. 2025Insertion Language Models: Sequence Generation with Arbitrary-Position Insertionscited by 11
  7. 2020Insertion-Deletion Transformercited by 9
  8. 2025XLM: A Python package for non-autoregressive language modelscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes
  2. 2026Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models
  3. 2025Dream 7B: Diffusion Large Language Models
  4. 2026Hierarchical Continuous Diffusion Language Models
  5. 2025Theoretical Benefit and Limitation of Diffusion Language Model
  6. 2026Multi-Mask Diffusion Language Models for Few-Step Generation
  7. 2026Diffusion Language Models: An Experimental Analysis
  8. 2026Masked Language Flow Models