mlsanityData to 7 Oct 2026

A Survey on Large Language Models for Code Generation

arXiv 2406.00515 ↗First posted 1 Jun 2024cs.CL cs.AI cs.SEcited by 646

Not on the map. Why?

Cited by 122 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) have garnered remarkable advancements across diverse code-related tasks, known as Code LLMs, particularly in code generation that generates source code with LLM from natural language descriptions. This burgeoning field has captured significant interest from both academic researchers and industry professionals due to its practical significance in software development, e.g., GitHub Copilot. Despite the active exploration of LLMs for a variety of code tasks, either from the perspective of natural language processing (NLP) or software engineering (SE) or both, there is a noticeable absence of a comprehensive and up-to-date literature review dedicated to LLM for code generation. In this survey, we aim to bridge this gap by providing a systematic literature review that serves as a valuable reference for researchers investigating the cutting-edge progress in LLMs for code generation. We introduce a taxonomy to categorize and discuss the recent developments in LLMs for code generation, covering aspects such as data curation, latest advances, performance evaluation, ethical implications, environmental impact, and real-world applications. In addition, we present a historical overview of the evolution of LLMs for code generation and offer an empirical comparison using the HumanEval, MBPP, and BigCodeBench benchmarks across various levels of difficulty and types of programming tasks to highlight the progressive enhancements in LLM capabilities for code generation. We identify critical challenges and promising opportunities regarding the gap between academia and practical development. Furthermore, we have established a dedicated resource GitHub page (https://github.com/juyongjiang/CodeLLMSurvey) to continuously document and disseminate the most recent advances in the field.

Key facts. Cited by 646 papers in the map (data through 7 Oct 2026). Builds on 178 of its 289 references that are in the map. First posted on arXiv 1 Jun 2024.

Builds on

178 of 289 references.

References · 178

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2017Proximal Policy Optimization Algorithmscited by 14,441
  9. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  10. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  11. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  12. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
38 more
  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2016Layer Normalizationcited by 5,579
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  13. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  14. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  15. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  16. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  17. 2021Program Synthesis with Large Language Modelscited by 3,509
  18. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  19. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  20. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  21. 2023Qwen Technical Reportcited by 3,273
  22. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  23. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  24. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  25. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  26. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  27. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  28. 2020Extracting Training Data from Large Language Modelscited by 2,144
  29. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  30. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  31. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  32. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  33. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  34. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  35. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  36. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  37. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  38. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154

50 most cited of 178 All 178 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 68; 2025: 337; 2026: 241.

646 papers.

Sort:

By area:

  1. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  2. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  3. 2025Video models are zero-shot learners and reasonerscited by 208
  4. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
  5. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  6. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  7. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 86
  8. 2024Non-Determinism of "Deterministic" LLM Settingscited by 84
  9. 2024Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Surveycited by 82
  10. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agentscited by 80
  11. 2025ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoningcited by 70
  12. 2024A Systematic Survey on Large Language Models for Algorithm Designcited by 65
38 more
  1. 2024CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databasescited by 62
  2. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  3. 2025Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operationscited by 46
  4. 2025WebWeaver: Structuring Web-Scale Evidence with Dynamic Outlines for Open-Ended Deep Researchcited by 44
  5. 2025ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMscited by 42
  6. 2025CudaForge: An Agent Framework with Hardware Feedback for CUDA Kernel Optimizationcited by 42
  7. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  8. 2024An Intelligent Agentic System for Complex Image Restoration Problemscited by 41
  9. 2024Meta Large Language Model Compiler: Foundation Models of Compiler Optimizationcited by 39
  10. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 39
  11. 2025CRANE: Reasoning with constrained LLM generationcited by 38
  12. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  13. 2025ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluationcited by 35
  14. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  15. 2025Type-Constrained Code Generation with Language Modelscited by 32
  16. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learningcited by 32
  17. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  18. 2025Enforcing Temporal Constraints for LLM Agentscited by 31
  19. 2024Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machinacited by 29
  20. 2025R1-Compress: Long Chain-of-Thought Compression via Chunk Compression and Searchcited by 29
  21. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Researchcited by 29
  22. 2025RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Codecited by 28
  23. 2025Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystemcited by 28
  24. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 27
  25. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 26
  26. 2025Cognitive Foundations for Reasoning and Their Manifestation in LLMscited by 26
  27. 2024Multi-Agent Large Language Models for Conversational Task-Solvingcited by 24
  28. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  29. 2025Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMscited by 24
  30. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exitingcited by 23
  31. 2025Premise-Augmented Reasoning Chains Improve Error Identification in Math reasoning with LLMscited by 22
  32. 2025DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environmentscited by 22
  33. 2025MLZero: A Multi-Agent System for End-to-end Machine Learning Automationcited by 22
  34. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  35. 2025Process-Supervised Reinforcement Learning for Code Generationcited by 21
  36. 2025SteerConf: Steering LLMs for Confidence Elicitationcited by 20
  37. 2025Exploring LLM Agents for Cleaning Tabular Machine Learning Datasetscited by 20
  38. 2025A Study of LLMs' Preferences for Libraries and Programming Languagescited by 20

50 most cited of 646 All 646 (JSON)