mlsanityData to 14 Oct 2026

InCoder: A Generative Model for Code Infilling and Synthesis

arXiv 2204.05999 ↗First posted 12 Apr 2022cs.SE cs.CL cs.LGcited by 498

Not on the map. Why?

Cited by 37 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Code is seldom written in a single left-to-right pass and is instead repeatedly edited and refined. We introduce InCoder, a unified generative model that can perform program synthesis (via left-to-right generation) as well as editing (via infilling). InCoder is trained to generate code files from a large corpus of permissively licensed code, where regions of code have been randomly masked and moved to the end of each file, allowing code infilling with bidirectional context. Our model is the first generative model that is able to directly perform zero-shot code infilling, which we evaluate on challenging tasks such as type inference, comment generation, and variable re-naming. We find that the ability to condition on bidirectional context substantially improves performance on these tasks, while still performing comparably on standard program synthesis benchmarks in comparison to left-to-right only models pretrained at similar scale. The InCoder models and code are publicly released. https://sites.google.com/view/incoder-code-models

Key facts. Cited by 498 papers in the map (data through 14 Oct 2026). Builds on 57 of its 81 references that are in the map. First posted on arXiv 12 Apr 2022.

Builds on

57 of 81 references.

References · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
38 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,637
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  7. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  9. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  10. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  11. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  12. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  13. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  14. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  15. 2019Defending Against Neural Fake Newscited by 559
  16. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  17. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  18. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  19. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  20. 2019SPoC: Search-based Pseudocode to Codecited by 176
  21. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  22. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  23. 2016DeepCoder: Learning to Write Programscited by 149
  24. 2018SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repaircited by 145
  25. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  26. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  27. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 126
  28. 2019Insertion Transformer: Flexible Sequence Generation via Insertion Operationscited by 125
  29. 2020Enabling Language Models to Fill in the Blankscited by 122
  30. 2020Graph-based, Self-Supervised Program Repair from Diagnostic Feedbackcited by 94
  31. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  32. 2021Break-It-Fix-It: Unsupervised Learning for Program Repaircited by 71
  33. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  34. 2021HTLM: Hyper-Text Pre-Training and Prompting of Language Modelscited by 58
  35. 2022CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequencescited by 50
  36. 2019JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generationcited by 49
  37. 2020LambdaNet: Probabilistic Type Inference using Graph Neural Networkscited by 44
  38. 2021SpreadsheetCoder: Formula Prediction from Semi-structured Contextcited by 41

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 37; 2023: 150; 2024: 156; 2025: 97; 2026: 58.

498 papers.

Sort:

By area:

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2023A Survey of Large Language Modelscited by 2,576
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  8. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  11. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  12. 2023StarCoder: may the source be with you!cited by 799
38 more
  1. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  2. 2024A Survey on Large Language Models for Code Generationcited by 658
  3. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  4. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  5. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  6. 2022CodeT: Code Generation with Generated Testscited by 384
  7. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  8. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  9. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  10. 2023Tool Learning with Foundation Modelscited by 302
  11. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  12. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  13. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  14. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  15. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 247
  16. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 239
  17. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
  18. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  19. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 183
  20. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
  21. 2023SantaCoder: don't reach for the stars!cited by 170
  22. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 168
  23. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  24. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  25. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 155
  26. 2022DocPrompting: Generating Code by Retrieving the Docscited by 153
  27. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 152
  28. 2023Emergent and Predictable Memorization in Large Language Modelscited by 148
  29. 2023Who Wrote this Code? Watermarking for Code Generationcited by 141
  30. 2022Learning by Distilling Contextcited by 138
  31. 2022Large Language Models Meet NL2Code: A Surveycited by 138
  32. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 138
  33. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 136
  34. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 134
  35. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  36. 2022Few-shot training LLMs for project-specific code-summarizationcited by 129
  37. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  38. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 125

50 most cited of 498 All 498 (JSON)