mlsanityData to 7 Oct 2026

Efficient Training of Language Models to Fill in the Middle

arXiv 2207.14255 ↗First posted 28 Jul 2022cs.CLcited by 248

Not on the map. Why?

Cited by 28 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We show that autoregressive language models can learn to infill text after we apply a straightforward transformation to the dataset, which simply moves a span of text from the middle of a document to its end. While this data augmentation has garnered much interest in recent years, we provide extensive evidence that training models with a large fraction of data transformed in this way does not harm the original left-to-right generative capability, as measured by perplexity and sampling evaluations across a wide range of scales. Given the usefulness, simplicity, and efficiency of training models to fill-in-the-middle (FIM), we suggest that future autoregressive language models be trained with FIM by default. To this end, we run a series of ablations on key hyperparameters, such as the data transformation frequency, the structure of the transformation, and the method of selecting the infill span. We use these ablations to prescribe strong default settings and best practices to train FIM models. We have released our best infilling model trained with best practices in our API, and release our infilling benchmarks to aid future research.

Key facts. Cited by 248 papers in the map (data through 7 Oct 2026). Builds on 44 of its 56 references that are in the map. First posted on arXiv 28 Jul 2022.

Builds on

44 of 56 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
32 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  11. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  12. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  13. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  14. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  15. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  16. 2018QuAC : Question Answering in Contextcited by 379
  17. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  18. 2021Scaling Laws for Transfercited by 265
  19. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  20. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  21. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  22. 2019Insertion Transformer: Flexible Sequence Generation via Insertion Operationscited by 123
  23. 2020Enabling Language Models to Fill in the Blankscited by 121
  24. 2018MaskGAN: Better Text Generation via Filling in the______cited by 89
  25. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  26. 2021HTLM: Hyper-Text Pre-Training and Prompting of Language Modelscited by 58
  27. 2019Insertion-based Decoding with automatically Inferred Generation Ordercited by 43
  28. 2020Blank Language Modelscited by 43
  29. 2021ReasonBERT: Pre-trained to Reason with Distant Supervisioncited by 22
  30. 2019KERMIT: Generative Insertion-Based Modeling for Sequencescited by 21
  31. 2022On the Role of Bidirectionality in Language Model Pre-Trainingcited by 15
  32. 2019TIGS: An Inference Algorithm for Text Infilling with Gradient Searchcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 47; 2024: 68; 2025: 74; 2026: 51.

248 papers.

Sort:

By area:

  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  3. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  4. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  5. 2025Large Language Diffusion Modelscited by 872
  6. 2023Textbooks Are All You Needcited by 561
  7. 2023The Falcon Series of Open Language Modelscited by 526
  8. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  9. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  10. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  11. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  12. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
38 more
  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  3. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 208
  4. 2022Continuous diffusion for categorical datacited by 202
  5. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  6. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  7. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  8. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  9. 2023SantaCoder: don't reach for the stars!cited by 169
  10. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Proverscited by 161
  11. 2025Qwen2.5-1M Technical Reportcited by 158
  12. 2023Protein Design with Guided Discrete Diffusioncited by 154
  13. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 149
  14. 2024CodeGemma: Open Code Models Based on Gemmacited by 145
  15. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  16. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  17. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  18. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 124
  19. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  20. 2023Who Wrote this Code? Watermarking for Code Generationcited by 116
  21. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  22. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  23. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 81
  24. 2023LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Modelscited by 80
  25. 2023Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programscited by 79
  26. 2025Path Planning for Masked Diffusion Model Samplingcited by 79
  27. 2023AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZcited by 74
  28. 2023IconShop: Text-Guided Vector Icon Synthesis with Autoregressive Transformerscited by 73
  29. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  30. 2023TESS: Text-to-Text Self-Conditioned Simplex Diffusioncited by 63
  31. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  32. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
  33. 2024Repoformer: Selective Retrieval for Repository-Level Code Completioncited by 55
  34. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
  35. 2023RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repaircited by 50
  36. 2024Reverse Training to Nurse the Reversal Cursecited by 48
  37. 2025Any-Order Flexible Length Masked Diffusioncited by 48
  38. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47

50 most cited of 248 All 248 (JSON)