mlsanityData to 7 Oct 2026

A Systematic Evaluation of Large Language Models of Code

arXiv 2202.13169 ↗First posted 26 Feb 2022cs.PL cs.CLcited by 360

Not on the map. Why?

Cited by 22 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LMs) of code have recently shown tremendous promise in completing code and synthesizing code from natural language descriptions. However, the current state-of-the-art code LMs (e.g., Codex (Chen et al., 2021)) are not publicly available, leaving many questions about their model and data design decisions. We aim to fill in some of these blanks through a systematic evaluation of the largest existing models: Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot, across various programming languages. Although Codex itself is not open-source, we find that existing open-source models do achieve close results in some programming languages, although targeted mainly for natural language modeling. We further identify an important missing piece in the form of a large open-source model trained exclusively on a multi-lingual corpus of code. We release a new model, PolyCoder, with 2.7B parameters based on the GPT-2 architecture, which was trained on 249GB of code across 12 programming languages on a single machine. In the C programming language, PolyCoder outperforms all models including Codex. Our trained models are open-source and publicly available at https://github.com/VHellendoorn/Code-LMs, which enables future research and application in this area.

Key facts. Cited by 360 papers in the map (data through 7 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 26 Feb 2022.

Builds on

22 of 37 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
10 more
  1. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  2. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  3. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  4. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  5. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  6. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 123
  7. 2021Language-Agnostic Representation Learning of Source Code from Structure and Contextcited by 63
  8. 2020Big Code != Big Vocabulary: Open-Vocabulary Models for Source Codecited by 53
  9. 2015Program Synthesis using Natural Languagecited by 41
  10. 2019An Empirical Study of C++ Vulnerabilities in Crowd-Sourced Code Examplescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 31; 2023: 113; 2024: 132; 2025: 67; 2026: 17.

360 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  4. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  5. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  6. 2023Teaching Large Language Models to Self-Debugcited by 963
  7. 2023StarCoder: may the source be with you!cited by 790
  8. 2024A Survey on Large Language Models for Code Generationcited by 646
  9. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  10. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  11. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  12. 2022CodeT: Code Generation with Generated Testscited by 369
38 more
  1. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  2. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  3. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  4. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  5. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  6. 2023Using an LLM to Help With Code Understandingcited by 222
  7. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  8. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  9. 2022Can Foundation Models Wrangle Your Data?cited by 173
  10. 2023SantaCoder: don't reach for the stars!cited by 169
  11. 2024The Effect of Sampling Temperature on Problem Solving in Large Language Modelscited by 168
  12. 2023A Categorical Archive of ChatGPT Failurescited by 167
  13. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  14. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  15. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  16. 2022DocPrompting: Generating Code by Retrieving the Docscited by 151
  17. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 149
  18. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  19. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  20. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  21. 2023Large Language Models for Robotics: A Surveycited by 132
  22. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  23. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 128
  24. 2023Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPTcited by 125
  25. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110
  26. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  27. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  28. 2023Learning Performance-Improving Code Editscited by 103
  29. 2023A Survey of Text Watermarking in the Era of Large Language Modelscited by 102
  30. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  31. 2023DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detectioncited by 98
  32. 2023Fuzz4All: Universal Fuzzing with Large Language Modelscited by 88
  33. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  34. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  35. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  36. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  37. 2024CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needscited by 81
  38. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77

50 most cited of 360 All 360 (JSON)