mlsanityData to 7 Oct 2026

Enabling Language Models to Fill in the Blanks

arXiv 2005.05339 ↗First posted 11 May 2020cs.CL cs.AI cs.LGcited by 121

Not on the map. Why?

Abstract (arXiv)

We present a simple approach for text infilling, the task of predicting missing spans of text at any position in a document. While infilling could enable rich functionality especially for writing assistance tools, more attention has been devoted to language modeling---a special case of infilling where text is predicted at the end of a document. In this paper, we aim to extend the capabilities of language models (LMs) to the more general task of infilling. To this end, we train (or fine-tune) off-the-shelf LMs on sequences containing the concatenation of artificially-masked text and the text which was masked. We show that this approach, which we call infilling by language modeling, can enable LMs to infill entire sentences effectively on three different domains: short stories, scientific abstracts, and lyrics. Furthermore, we show that humans have difficulty identifying sentences infilled by our approach as machine-generated in the domain of short stories.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 34; 2023: 19; 2024: 23; 2025: 12; 2026: 5.

121 papers.

Sort:

By area:

  1. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  2. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  3. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  4. 2023Augmented Language Models: a Surveycited by 398
  5. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  6. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  7. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  8. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 142
  9. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  10. 2021COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemiccited by 100
  11. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
  12. 2023Amortizing intractable inference in large language modelscited by 94
38 more
  1. 2022PEER: A Collaborative Language Modelcited by 86
  2. 2022Self-Training: A Surveycited by 85
  3. 2023Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programscited by 79
  4. 2024A Design Space for Intelligent and Interactive Writing Assistantscited by 76
  5. 2022Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writingcited by 71
  6. 2022Data Feedback Loops: Model-driven Amplification of Dataset Biasescited by 59
  7. 2021Prompting Contrastive Explanations for Commonsense Reasoning Taskscited by 45
  8. 2022A Langevin-like Sampler for Discrete Distributionscited by 42
  9. 2022Faking Fake News for Real Fake News Detection: Propaganda-loaded Training Data Generationcited by 41
  10. 2022Attribution and Obfuscation of Neural Text Authorship: A Data Mining Perspectivecited by 40
  11. 2022Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infillingcited by 38
  12. 2023Anticipatory Music Transformercited by 35
  13. 2024Adaptable Logical Control for Large Language Modelscited by 30
  14. 2021Self-Supervised Euphemism Detection and Identification for Content Moderationcited by 24
  15. 2022Phrase-level Textual Adversarial Attack with Label Preservationcited by 23
  16. 2021Conformal prediction for text infilling and part-of-speech predictioncited by 22
  17. 2023Meet in the Middle: A New Pre-training Paradigmcited by 22
  18. 2023Does Human Collaboration Enhance the Accuracy of Identifying LLM-Generated Deepfake Texts?cited by 21
  19. 2023Guiding Language Models of Code with Global Context using Monitorscited by 20
  20. 2022Language modeling via stochastic processescited by 18
  21. 2022Necessity and Sufficiency for Explaining Text Classifiers: A Case Study in Hate Speech Detectioncited by 18
  22. 2023Towards Imperceptible Document Manipulations against Neural Ranking Modelscited by 18
  23. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  24. 2021Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluationcited by 16
  25. 2021C5T5: Controllable Generation of Organic Molecules with Transformerscited by 15
  26. 2021Inspiration through Observation: Demonstrating the Influence of Automatically Generated Text on Creative Writingcited by 14
  27. 2021Variable-Length Music Score Infilling via XLNet and Musically Specialized Positional Encodingcited by 14
  28. 2021Goal-Directed Story Generation: Augmenting Generative Language Models with Reinforcement Learningcited by 14
  29. 2022Robust Preference Learning for Storytelling via Contrastive Reinforcement Learningcited by 14
  30. 2021Euphemistic Phrase Detection by Masked Language Modelcited by 13
  31. 2021MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluationcited by 13
  32. 2023FiLM: Fill-in Language Models for Any-Order Generationcited by 13
  33. 2024TrajGPT: Controlled Synthetic Trajectory Generation Using a Multitask Transformer-Based Spatiotemporal Modelcited by 13
  34. 2021IGA : An Intent-Guided Authoring Assistantcited by 12
  35. 2023Mask-then-Fill: A Flexible and Effective Data Augmentation Framework for Event Extractioncited by 12
  36. 2023Storyfier: Exploring Vocabulary Learning Support with Text Generation Modelscited by 12
  37. 2024Fakes of Varying Shades: How Warning Affects Human Perception and Engagement Regarding LLM Hallucinationscited by 12
  38. 2021A Computational Framework for Slang Generationcited by 11

50 most cited of 121 All 121 (JSON)