# Text Infilling

arXiv: https://arxiv.org/abs/1901.00158 · HTML: https://mlsanity.com/paper/1901.00158/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jan 2019.

## Built on by (most cited first, 30 of 32)

- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics](https://mlsanity.com/paper/2202.11705/) (2022)
- [Amortizing intractable inference in large language models](https://mlsanity.com/paper/2310.04363/) (2023)
- [Amortizing intractable inference in diffusion models for vision, language, and control](https://mlsanity.com/paper/2405.20971/) (2024)
- [DeepStruct: Pretraining of Language Models for Structure Prediction](https://mlsanity.com/paper/2205.10475/) (2022)
- [Anticipatory Music Transformer](https://mlsanity.com/paper/2306.08620/) (2023)
- [Visual Abductive Reasoning](https://mlsanity.com/paper/2203.14040/) (2022)
- [Controllable Summarization with Constrained Markov Decision Process](https://mlsanity.com/paper/2108.03405/) (2021)
- [Quality Estimation without Human-labeled Data](https://mlsanity.com/paper/2102.04020/) (2021)
- [Variable-Length Music Score Infilling via XLNet and Musically Specialized Positional Encoding](https://mlsanity.com/paper/2108.05064/) (2021)
- [Generative Prompt Tuning for Relation Classification](https://mlsanity.com/paper/2210.12435/) (2022)
- [FiLM: Fill-in Language Models for Any-Order Generation](https://mlsanity.com/paper/2310.09930/) (2023)
- [IGA : An Intent-Guided Authoring Assistant](https://mlsanity.com/paper/2104.07000/) (2021)
- [GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation](https://mlsanity.com/paper/2211.10330/) (2022)
- [Show Me How To Revise: Improving Lexically Constrained Sentence Generation with XLNet](https://mlsanity.com/paper/2109.05797/) (2021)
- [Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling](https://mlsanity.com/paper/2307.08074/) (2023)
- [Language Model Augmented Relevance Score](https://mlsanity.com/paper/2108.08485/) (2021)
- [Controlling Text Edition by Changing Answers of Specific Questions](https://mlsanity.com/paper/2105.11018/) (2021)
- [Chinese Idiom Paraphrasing](https://mlsanity.com/paper/2204.07555/) (2022)
- [Controllable Fake Document Infilling for Cyber Deception](https://mlsanity.com/paper/2210.09917/) (2022)
- [Self-Infilling Code Generation](https://mlsanity.com/paper/2311.17972/) (2023)
- [Towards a Universal NLG for Dialogue Systems and Simulators with Future Bridging](https://mlsanity.com/paper/2105.10267/) (2021)
- [The Case for a Single Model that can Both Generate Continuations and Fill in the Blank](https://mlsanity.com/paper/2206.04812/) (2022)
- [Interactive Editing for Text Summarization](https://mlsanity.com/paper/2306.03067/) (2023)
- [GraphCheck: Multipath Fact-Checking with Entity-Relationship Graphs](https://mlsanity.com/paper/2502.20785/) (2025)
- [A Reinforcement Learning-Driven Transformer GAN for Molecular Generation](https://mlsanity.com/paper/2503.12796/) (2025)
- [Diffusion LMs Can Approximate Optimal Infilling Lengths Implicitly](https://mlsanity.com/paper/2602.00476/) (2026)
- [Advancing Semi-Supervised Learning for Automatic Post-Editing: Data-Synthesis by Mask-Infilling with Erroneous Terms](https://mlsanity.com/paper/2204.03896/) (2022)
- [Scheduling Thoughts: Learning the Order of Thought in Diffusion Language Models](https://mlsanity.com/paper/2606.23567/) (2026)
- [SC-Ques: A Sentence Completion Question Dataset for English as a Second Language Learners](https://mlsanity.com/paper/2206.12036/) (2022)

## Abstract (arXiv)

Recent years have seen remarkable progress of text generation in different contexts, such as the most common setting of generating text from scratch, and the emerging paradigm of retrieval-and-rewriting. Text infilling, which fills missing text portions of a sentence or paragraph, is also of numerous use in real life, yet is under-explored. Previous work has focused on restricted settings by either assuming single word per missing portion or limiting to a single missing portion to the end of the text. This paper studies the general task of text infilling, where the input text can have an arbitrary number of portions to be filled, each of which may require an arbitrary unknown number of tokens. We study various approaches for the task, including a self-attention model with segment-aware position encoding and bidirectional context modeling. We create extensive supervised data by masking out text with varying strategies. Experiments show the self-attention model greatly outperforms others, creating a strong baseline for future research.