mlsanityData to 14 Oct 2026

Program Synthesis with Large Language Models

arXiv 2108.07732 ↗First posted 16 Aug 2021cs.PL cs.LGcited by 3,637

Not on the map. Why?

Cited by 606 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper explores the limits of the current generation of large language models for program synthesis in general purpose programming languages. We evaluate a collection of such models (with between 244M and 137B parameters) on two new benchmarks, MBPP and MathQA-Python, in both the few-shot and fine-tuning regimes. Our benchmarks are designed to measure the ability of these models to synthesize short Python programs from natural language descriptions. The Mostly Basic Programming Problems (MBPP) dataset contains 974 programming tasks, designed to be solvable by entry-level programmers. The MathQA-Python dataset, a Python version of the MathQA benchmark, contains 23914 problems that evaluate the ability of the models to synthesize code from more complex text. On both datasets, we find that synthesis performance scales log-linearly with model size. Our largest models, even without finetuning on a code dataset, can synthesize solutions to 59.6 percent of the problems from MBPP using few-shot learning with a well-designed prompt. Fine-tuning on a held-out portion of the dataset improves performance by about 10 percentage points across most model sizes. On the MathQA-Python dataset, the largest fine-tuned model achieves 83.8 percent accuracy. Going further, we study the model's ability to engage in dialog about code, incorporating human feedback to improve its solutions. We find that natural language feedback from a human halves the error rate compared to the model's initial prediction. Additionally, we conduct an error analysis to shed light on where these models fall short and what types of programs are most difficult to generate. Finally, we explore the semantic grounding of these models by fine-tuning them to predict the results of program execution. We find that even our best models are generally unable to predict the output of a program given a specific input.

Key facts. Cited by 3,637 papers in the map (data through 14 Oct 2026). Builds on 56 of its 106 references that are in the map. First posted on arXiv 16 Aug 2021.

Builds on

56 of 106 references.

References · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2018Deep contextualized word representationscited by 2,290
  9. 2020Extracting Training Data from Large Language Modelscited by 2,189
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  11. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  12. 2021Measuring Coding Challenge Competence With APPScited by 902
38 more
  1. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  2. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  3. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  4. 2014Neural Turing Machinescited by 644
  5. 2020Unsupervised Translation of Programming Languagescited by 263
  6. 2018code2vec: Learning Distributed Representations of Codecited by 250
  7. 2020IntelliCode Compose: Code Generation Using Transformercited by 227
  8. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 221
  9. 2017Learning to Represent Programs with Graphscited by 220
  10. 2015Semi-supervised Sequence Learningcited by 198
  11. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  12. 2019SPoC: Search-based Pseudocode to Codecited by 176
  13. 2016Exploring the Limits of Language Modelingcited by 173
  14. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 161
  15. 2016DeepCoder: Learning to Write Programscited by 149
  16. 2018SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repaircited by 145
  17. 2020You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completioncited by 131
  18. 2017RobustFill: Neural Program Learning under Noisy I/Ocited by 126
  19. 2015Neural GPUs Learn Algorithmscited by 121
  20. 2014Learning to Executecited by 106
  21. 2018Mapping Language to Code in Programmatic Contextcited by 104
  22. 2020Graph-based, Self-Supervised Program Repair from Diagnostic Feedbackcited by 94
  23. 2019Write, Execute, Assess: Program Synthesis with a REPLcited by 93
  24. 2016A Convolutional Attention Network for Extreme Summarization of Source Codecited by 91
  25. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  26. 2016Latent Predictor Networks for Code Generationcited by 73
  27. 2020Code Prediction by Feeding Trees to Transformerscited by 70
  28. 2021Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Taskscited by 70
  29. 2018DeepBugs: A Learning Approach to Name-based Bug Detectioncited by 65
  30. 2020Big Code != Big Vocabulary: Open-Vocabulary Models for Source Codecited by 54
  31. 2020LambdaNet: Probabilistic Type Inference using Graph Neural Networkscited by 44
  32. 2020Learning to Execute Programs with Instruction Pointer Attention Graph Neural Networkscited by 35
  33. 2015Neural Random-Access Machinescited by 34
  34. 2019TypeWriter: Neural Type Prediction with Search-based Validationcited by 32
  35. 2020BUSTLE: Bottom-Up Program Synthesis Through Learning-Guided Explorationcited by 30
  36. 2014Learning Natural Coding Conventionscited by 29
  37. 2019Learning to Fix Build Errors with Graph2Diff Neural Networkscited by 26
  38. 2021Programming Puzzlescited by 25

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 88; 2023: 278; 2024: 709; 2025: 1099; 2026: 1446.

3,637 papers.

Sort:

By area:

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2023Mistral 7Bcited by 3,164
38 more
  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  3. 2024Qwen2 Technical Reportcited by 2,120
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  8. 2025Gemma 3 Technical Reportcited by 1,824
  9. 2024Mixtral of Expertscited by 1,823
  10. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  13. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  14. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  15. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  16. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  17. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  18. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  19. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  20. 2023PaLM 2 Technical Reportcited by 1,162
  21. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  22. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  23. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  24. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  25. 2023Teaching Large Language Models to Self-Debugcited by 988
  26. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  27. 2025Large Language Diffusion Modelscited by 912
  28. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  29. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  30. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  31. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  32. 2024DeepSeek-V3 Technical Reportcited by 811
  33. 2023StarCoder: may the source be with you!cited by 799
  34. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  35. 2025Qwen2.5-Omni Technical Reportcited by 736
  36. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  37. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  38. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723

50 most cited of 3,637 All 3,637 (JSON)