mlsanityData to 14 Oct 2026

Self-Infilling Code Generation

arXiv 2311.17972 ↗First posted 29 Nov 2023cs.PL cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

This work introduces self-infilling code generation, a general framework that incorporates infilling operations into auto-regressive decoding. Our approach capitalizes on the observation that recent infilling-capable code language models can self-infill: whereas infilling operations aim to fill in the middle based on a predefined prefix and suffix, self-infilling sequentially generates both such surrounding context and the infilled content. We utilize this capability to introduce novel interruption and looping mechanisms in conventional decoding, evolving it into a non-monotonic process. Interruptions allow for postponing the generation of specific code until a definitive suffix is established, enhancing control over the output. Meanwhile, the looping mechanism, which leverages the complementary nature of self-infilling and left-to-right decoding, can iteratively update and synchronize each piece of generation cyclically. Extensive experiments are conducted to demonstrate that our proposed decoding process is effective in enhancing both regularity and quality across several code generation benchmarks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 59 of its 77 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

59 of 77 references.

References · 59

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  10. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  11. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  12. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
38 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  9. 2023PaLM 2 Technical Reportcited by 1,162
  10. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  11. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  12. 2023Teaching Large Language Models to Self-Debugcited by 988
  13. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  14. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  15. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  16. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  17. 2023StarCoder: may the source be with you!cited by 799
  18. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  19. 2022PAL: Program-aided Language Modelscited by 652
  20. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  21. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  22. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  23. 2022UL2: Unifying Language Learning Paradigmscited by 295
  24. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  25. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 239
  26. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  27. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  28. 2023SantaCoder: don't reach for the stars!cited by 170
  29. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 170
  30. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  31. 2019Insertion Transformer: Flexible Sequence Generation via Insertion Operationscited by 125
  32. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  33. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 125
  34. 2020Enabling Language Models to Fill in the Blankscited by 122
  35. 2022Coder Reviewer Reranking for Code Generationcited by 84
  36. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 72
  37. 2021HTLM: Hyper-Text Pre-Training and Prompting of Language Modelscited by 58
  38. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 57

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 1.

3 papers.

  1. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 274
  2. 2024Non-myopic Generation of Language Models for Reasoning and Planningcited by 26
  3. 2026Autoregressive, Yet Revisable: In Decoding Revision for Secure Code Generationcited by 1