mlsanityData to 7 Oct 2026

Large Language Models Meet NL2Code: A Survey

arXiv 2212.09420 ↗First posted 19 Dec 2022cs.SE cs.AI cs.CLcited by 135

Not on the map. Why?

Abstract (arXiv)

The task of generating code from a natural language description, or NL2Code, is considered a pressing and significant challenge in code intelligence. Thanks to the rapid development of pre-training techniques, surging large language models are being proposed for code, sparking the advances in NL2Code. To facilitate further research and applications in this field, in this paper, we present a comprehensive survey of 27 existing large language models for NL2Code, and also review benchmarks and metrics. We provide an intuitive comparison of all existing models on the HumanEval benchmark. Through in-depth observation and analysis, we provide some insights and conclude that the key factors contributing to the success of large language models for NL2Code are "Large Size, Premium Data, Expert Tuning". In addition, we discuss challenges and opportunities regarding the gap between models and humans. We also create a website https://nl2code.github.io to track the latest progress through crowd-sourcing. To the best of our knowledge, this is the first survey of large language models for NL2Code, and we believe it will contribute to the ongoing development of the field.

Key facts. Cited by 135 papers in the map (data through 7 Oct 2026). Builds on 63 of its 123 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

63 of 123 references.

References · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
38 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  2. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  3. 2021Measuring Coding Challenge Competence With APPScited by 879
  4. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  5. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  7. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  8. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  9. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  10. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  11. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  12. 2022Self-critiquing models for assisting human evaluatorscited by 372
  13. 2022CodeT: Code Generation with Generated Testscited by 369
  14. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  15. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  16. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  17. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  18. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  19. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  20. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 220
  21. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
  22. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  23. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  24. 2022Evaluating the Text-to-SQL Capabilities of Large Language Modelscited by 101
  25. 2018Improving Automatic Source Code Summarization via Deep Reinforcement Learningcited by 99
  26. 2022Automating Code Review Activities by Large-Scale Pre-trainingcited by 98
  27. 2021A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Levelcited by 96
  28. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  29. 2022Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Bookcited by 84
  30. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  31. 2022SELFIES and the future of molecular string representationscited by 76
  32. 2021Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Taskscited by 70
  33. 2022Repair Is Nearly Generation: Multilingual Program Repair with LLMscited by 68
  34. 2020Deep Learning for Source Code Modeling and Generation: Models, Applications and Challengescited by 55
  35. 2022XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligencecited by 54
  36. 2022When Language Model Meets Private Librarycited by 54
  37. 2019JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generationcited by 49
  38. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 22; 2024: 59; 2025: 41; 2026: 13.

135 papers.

Sort:

By area:

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  3. 2024A Survey on Large Language Models for Code Generationcited by 646
  4. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  5. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  6. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  7. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  8. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 134
  9. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  10. 2024CodeR: Issue Resolving with Multi-Agent and Task Graphscited by 89
  11. 2023The Robots are Here: Navigating the Generative AI Revolution in Computing Educationcited by 85
  12. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
38 more
  1. 2023AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZcited by 74
  2. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  3. 2024Can Language Models Solve Olympiad Programming?cited by 70
  4. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  5. 2024A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languagescited by 65
  6. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 59
  7. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  8. 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 49
  9. 2024Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMscited by 48
  10. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  11. 2024LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generationcited by 45
  12. 2024Human-In-the-Loop Software Development Agentscited by 42
  13. 2024Are LLMs Ready for Real-World Materials Discovery?cited by 40
  14. 2024GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph-based Retrieval and Language Modelcited by 40
  15. 2024CYCLE: Learning to Self-Refine the Code Generationcited by 39
  16. 2024Mercury: A Code Efficiency Benchmark for Code Large Language Modelscited by 34
  17. 2024SWE-bench-java: A GitHub Issue Resolving Benchmark for Javacited by 34
  18. 2024MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMscited by 32
  19. 2025Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQLcited by 30
  20. 2023Emergent Representations of Program Semantics in Language Models Trained on Programscited by 28
  21. 2024CodeS: Natural Language to Code Repository via Multi-Layer Sketchcited by 28
  22. 2024The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmerscited by 28
  23. 2023BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Modelscited by 24
  24. 2024HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agentcited by 24
  25. 2024Helpful assistant or fruitful facilitator? Investigating how personas affect language model behaviorcited by 21
  26. 2024DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generationcited by 21
  27. 2025Process-Supervised Reinforcement Learning for Code Generationcited by 21
  28. 2024BudgetMLAgent: A Cost-Effective LLM Multi-Agent system for Automating Machine Learning Taskscited by 19
  29. 2024Prompt-based Code Completion via Multi-Retrieval Augmented Generationcited by 16
  30. 2024Code Simulation Challenges for Large Language Modelscited by 15
  31. 2024Beyond Functional Correctness: Investigating Coding Style Inconsistencies in Large Language Modelscited by 15
  32. 2023FRANC: A Lightweight Framework for High-Quality Code Generationcited by 14
  33. 2023Benchmarking Causal Study to Interpret Large Language Models for Source Codecited by 14
  34. 2025CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoningcited by 14
  35. 2024PromptIntern: Saving Inference Costs by Internalizing Recurrent Prompt during Large Language Model Fine-tuningcited by 13
  36. 2023Testing LLMs on Code Generation with Varying Levels of Prompt Specificitycited by 12
  37. 2024LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Surveycited by 12
  38. 2025Tokenization is Sensitive to Language Variationcited by 12

50 most cited of 135 All 135 (JSON)