mlsanityData to 7 Oct 2026

Evaluating Large Language Models Trained on Code

arXiv 2107.03374 ↗First posted 7 Jul 2021cs.LGcited by 8,053

Not on the map. Why?

Cited by 1,394 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.

Key facts. Cited by 8,053 papers in the map (data through 7 Oct 2026). Builds on 66 of its 130 references that are in the map. First posted on arXiv 7 Jul 2021.

Builds on

66 of 130 references.

References · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  11. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  12. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
38 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  13. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  14. 2021Measuring Coding Challenge Competence With APPScited by 879
  15. 2016Pixel Recurrent Neural Networkscited by 814
  16. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  17. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  18. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  19. 2014Neural Turing Machinescited by 630
  20. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  21. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  22. 2020Jukebox: A Generative Model for Musiccited by 535
  23. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  24. 2016Visual Dialogcited by 377
  25. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  26. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  27. 2020Unsupervised Translation of Programming Languagescited by 262
  28. 2015Semi-supervised Sequence Learningcited by 198
  29. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 177
  30. 2019SPoC: Search-based Pseudocode to Codecited by 176
  31. 2020Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defensescited by 171
  32. 2021Alignment of Language Agentscited by 167
  33. 2016DeepCoder: Learning to Write Programscited by 145
  34. 2017RobustFill: Neural Program Learning under Noisy I/Ocited by 124
  35. 2015Neural GPUs Learn Algorithmscited by 118
  36. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  37. 2020You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completioncited by 112
  38. 2018An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translationcited by 106

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 52; 2022: 304; 2023: 1005; 2024: 1625; 2025: 2372; 2026: 2695.

8,053 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2025Qwen3 Technical Reportcited by 8,195
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2024Qwen2.5 Technical Reportcited by 4,713
38 more
  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2023Mistral 7Bcited by 3,065
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2023A Survey of Large Language Modelscited by 2,537
  13. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  14. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  15. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  16. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  17. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  18. 2024Qwen2 Technical Reportcited by 2,076
  19. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  20. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  21. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  22. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  23. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  24. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  25. 2024Mixtral of Expertscited by 1,785
  26. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  27. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  28. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  29. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  30. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  31. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  32. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  33. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  34. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  35. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  36. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  37. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  38. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348

50 most cited of 8,053 All 8,053 (JSON)