# Blank Language Models

arXiv: https://arxiv.org/abs/2002.03079 · HTML: https://mlsanity.com/paper/2002.03079/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2020.

## Built on by (most cited first, 30 of 43)

- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022)
- [PEER: A Collaborative Language Model](https://mlsanity.com/paper/2208.11663/) (2022)
- [DeepStruct: Pretraining of Language Models for Structure Prediction](https://mlsanity.com/paper/2205.10475/) (2022)
- [Phrase-level Textual Adversarial Attack with Label Preservation](https://mlsanity.com/paper/2205.10710/) (2022)
- [Conformal prediction for text infilling and part-of-speech prediction](https://mlsanity.com/paper/2111.02592/) (2021)
- [DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas](https://mlsanity.com/paper/2602.01326/) (2026)
- [Inspiration through Observation: Demonstrating the Influence of Automatically Generated Text on Creative Writing](https://mlsanity.com/paper/2107.04007/) (2021)
- [Variable-Length Music Score Infilling via XLNet and Musically Specialized Positional Encoding](https://mlsanity.com/paper/2108.05064/) (2021)
- [FiLM: Fill-in Language Models for Any-Order Generation](https://mlsanity.com/paper/2310.09930/) (2023)
- [TrajGPT: Controlled Synthetic Trajectory Generation Using a Multitask Transformer-Based Spatiotemporal Model](https://mlsanity.com/paper/2411.04381/) (2024)
- [Machine-in-the-Loop Rewriting for Creative Image Captioning](https://mlsanity.com/paper/2111.04193/) (2021)
- [GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation](https://mlsanity.com/paper/2211.10330/) (2022)
- [BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models](https://mlsanity.com/paper/2404.04113/) (2024)
- [Crystal Transformer: Self-learning neural language model for Generative and Tinkering Design of Materials](https://mlsanity.com/paper/2204.11953/) (2022)
- [Probabilistic Generative Transformer Language models for Generative Design of Molecules](https://mlsanity.com/paper/2209.09406/) (2022)
- [On the Impact of Noise in Differentially Private Text Rewriting](https://mlsanity.com/paper/2501.19022/) (2025)
- [Language Model Augmented Relevance Score](https://mlsanity.com/paper/2108.08485/) (2021)
- [Toward Unified Controllable Text Generation via Regular Expression Instruction](https://mlsanity.com/paper/2309.10447/) (2023)
- [Nutri-bullets: Summarizing Health Studies by Composing Segments](https://mlsanity.com/paper/2103.11921/) (2021)
- [Chinese Idiom Paraphrasing](https://mlsanity.com/paper/2204.07555/) (2022)
- [Discovery of 2D materials using Transformer Network based Generative Design](https://mlsanity.com/paper/2301.05824/) (2023)
- [MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities](https://mlsanity.com/paper/2501.08648/) (2025)
- [Sentence-Permuted Paragraph Generation](https://mlsanity.com/paper/2104.07228/) (2021)
- [Nutribullets Hybrid: Multi-document Health Summarization](https://mlsanity.com/paper/2104.03465/) (2021)
- [Controllable Fake Document Infilling for Cyber Deception](https://mlsanity.com/paper/2210.09917/) (2022)
- [Replacing Language Model for Style Transfer](https://mlsanity.com/paper/2211.07343/) (2022)
- [Self-Infilling Code Generation](https://mlsanity.com/paper/2311.17972/) (2023)

## Abstract (arXiv)

We propose Blank Language Model (BLM), a model that generates sequences by dynamically creating and filling in blanks. The blanks control which part of the sequence to expand, making BLM ideal for a variety of text editing and rewriting tasks. The model can start from a single blank or partially completed text with blanks at specified locations. It iteratively determines which word to place in a blank and whether to insert new blanks, and stops generating when no blanks are left to fill. BLM can be efficiently trained using a lower bound of the marginal data likelihood. On the task of filling missing text snippets, BLM significantly outperforms all other baselines in terms of both accuracy and fluency. Experiments on style transfer and damaged ancient text restoration demonstrate the potential of this framework for a wide range of applications.