mlsanityData to 7 Oct 2026

CodeGen2: Lessons for Training LLMs on Programming and Natural Languages

arXiv 2305.02309 ↗First posted 3 May 2023cs.LGcited by 124

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable abilities in representation learning for program synthesis and understanding tasks. The quality of the learned representations appears to be dictated by the neural scaling laws as a function of the number of model parameters and observations, while imposing upper bounds on the model performance by the amount of available data and compute, which is costly. In this study, we attempt to render the training of LLMs for program synthesis more efficient by unifying four key components: (1) model architectures, (2) learning methods, (3) infill sampling, and, (4) data distributions. Specifically, for the model architecture, we attempt to unify encoder and decoder-based models into a single prefix-LM. For learning methods, (i) causal language modeling, (ii) span corruption, (iii) infilling are unified into a simple learning algorithm. For infill sampling, we explore the claim of a "free lunch" hypothesis. For data distributions, the effect of a mixture distribution and multi-epoch training of programming and natural languages on model performance is explored. We conduct a comprehensive series of empirical experiments on 1B LLMs, for which failures and successes of this exploration are distilled into five lessons. We will provide a final recipe for training and release CodeGen2 models in size 1B, 3.7B, 7B, and, 16B parameters, along with the training framework as open-source: https://github.com/salesforce/CodeGen.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Builds on 26 of its 31 references that are in the map. First posted on arXiv 3 May 2023.

Builds on

26 of 31 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
14 more
  1. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  2. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  4. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  6. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  7. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  8. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  9. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  10. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  11. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  12. 2023SantaCoder: don't reach for the stars!cited by 169
  13. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  14. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 55

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 25; 2024: 63; 2025: 33; 2026: 3.

124 papers.

Sort:

By area:

  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  3. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  4. 2024A Survey on Large Language Models for Code Generationcited by 646
  5. 2023Textbooks Are All You Needcited by 561
  6. 2024Large Language Models: A Surveycited by 506
  7. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  8. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  9. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  10. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 183
  11. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  12. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
38 more
  1. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  2. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
  3. 2024BetterV: Controlled Verilog Generation with Discriminative Guidancecited by 99
  4. 2024VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Toolcited by 85
  5. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  6. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  7. 2024Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLMcited by 63
  8. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  9. 2023CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modulescited by 58
  10. 2023Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repaircited by 55
  11. 2024Repoformer: Selective Retrieval for Repository-Level Code Completioncited by 55
  12. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  13. 2024AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learningcited by 53
  14. 2024Calibration and Correctness of Language Models for Codecited by 49
  15. 2023Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMscited by 45
  16. 2024Reasoning Runtime Behavior of a Program with LLM: How Far Are We?cited by 44
  17. 2024GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph-based Retrieval and Language Modelcited by 40
  18. 2024Finetuning Large Language Models for Vulnerability Detectioncited by 39
  19. 2024LeDex: Training LLMs to Better Self-Debug and Explain Codecited by 39
  20. 2023Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queriescited by 38
  21. 2024AutoVCoder: A Systematic Framework for Automated Verilog Code Generation using LLMscited by 36
  22. 2023SALLM: Security Assessment of Generated Codecited by 35
  23. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
  24. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 34
  25. 2024CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrievalcited by 33
  26. 2024AST-T5: Structure-Aware Pretraining for Code Generation and Understandingcited by 32
  27. 2024REPOFUSE: Repository-Level Code Completion with Fused Dual Contextcited by 32
  28. 2024LaMAGIC: Language-Model-based Topology Generation for Analog Integrated Circuitscited by 32
  29. 2023AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisationcited by 31
  30. 2024History, Development, and Principles of Large Language Models-An Introductory Surveycited by 30
  31. 2023Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Modelscited by 28
  32. 2024Between Lines of Code: Unraveling the Distinct Patterns of Machine and Human Programmerscited by 27
  33. 2023BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Modelscited by 24
  34. 2024NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctnesscited by 23
  35. 2023L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Modelscited by 22
  36. 2023LLM for SoC Security: A Paradigm Shiftcited by 22
  37. 2024CodeEditorBench: Evaluating Code Editing Capability of Large Language Modelscited by 22
  38. 2025QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checkscited by 22

50 most cited of 124 All 124 (JSON)