mlsanityData to 14 Oct 2026

Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applications

arXiv 2503.01245 ↗First posted 3 Mar 2025cs.SE cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated their remarkable capabilities in numerous fields. This survey focuses on how LLMs empower users, regardless of their technical background, to use human languages to automatically generate executable code. We begin with understanding LLMs' limitations and challenges in automated code generation. Subsequently, we review various fine-tuning techniques designed to enhance both the performance and adaptability of LLMs in code generation tasks. We then review the existing metrics and benchmarks for evaluations to assess model performance based on fine-tuning techniques. Finally, we explore the applications of LLMs (e.g. CodeLlama, GitHub Copilot, ToolGen) in code generation tasks to illustrate their roles and functionalities. This survey provides a comprehensive overview of LLMs for code generation, helps researchers in diverse fields better understand the current state-of-the-art technologies, and offers the potential of effectively leveraging LLMs for code generation tasks.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 32 of its 60 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

32 of 60 references.

References · 32

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  2. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  3. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  4. 2023Large Language Modelscited by 780
  5. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  6. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 282
  7. 2023Planning with Large Language Models for Code Generationcited by 182
  8. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 168
  9. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  10. 2023Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issuescited by 79
  11. 2023CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Modelscited by 55
  12. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
20 more
  1. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 53
  2. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 52
  3. 2024LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generationcited by 45
  4. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 43
  5. 2023ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarificationcited by 38
  6. 2024Towards Translating Real-World Code with LLMs: A Study of Translating to Rustcited by 35
  7. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 34
  8. 2024Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generationcited by 30
  9. 2024Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarkingcited by 27
  10. 2023AceCoder: Utilizing Existing Code to Enhance Code Generationcited by 25
  11. 2023Uncovering and Quantifying Social Biases in Code Generationcited by 25
  12. 2024The Landscape and Challenges of HPC Research and LLMscited by 22
  13. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 21
  14. 2024Benchmarks and Metrics for Evaluations of Code Generation: A Critical Reviewcited by 20
  15. 2024Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruningcited by 20
  16. 2024Enhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of LLMscited by 17
  17. 2024Exploring Multi-Lingual Bias of Large Code Models in Code Generationcited by 17
  18. 2024Comparing large language models and human programmers for generating programming codecited by 9
  19. 2024LLM Agents Improve Semantic Code Searchcited by 7
  20. 2024Enhancing Source Code Security with LLMs: Demystifying The Challenges and Generating Reliable Repairscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 14.

26 papers.

Sort:

By area:

  1. 2025Uncertainty-Guided Chain-of-Thought for Code Generation with LLMscited by 32
  2. 2025When LLMs Meet API Documentation: Can Retrieval Augmentation Aid Code Generation Just as It Helps Developers?cited by 13
  3. 2026Token-Level LLM Collaboration via FusionRoutecited by 9
  4. 2025Is Your Benchmark Still Useful? Dynamic Benchmarking for Code Language Modelscited by 6
  5. 2026OpInf-LLM: Parametric PDE Solving with LLMs via Operator Inferencecited by 5
  6. 2026Specification-Based Code-Text-Code Reengineering for LLM-Mediated Software Evolutioncited by 5
  7. 2025Code Reasoning for Software Engineering Tasks: A Survey and A Call to Actioncited by 4
  8. 2026ACE: Self-Evolving LLM Coding Framework via Adversarial Unit Test Generation and Preference Optimizationcited by 4
  9. 2026LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Reviewcited by 3
  10. 2026VeriContest: A Competitive-Programming Benchmark for Verifiable Code Generationcited by 3
  11. 2025WorkTeam: Constructing Workflows from Natural Language with Multi-Agentscited by 2
  12. 2025DataSage: Multi-agent Collaboration for Insight Discovery with External Knowledge Retrieval, Multi-role Debating, and Multi-path Reasoningcited by 2
14 more
  1. 2026Enhancing Reliability in LLM-Based Secure Code Generationcited by 2
  2. 2025OnRL-RAG: Real-Time Personalized Mental Health Dialogue Systemcited by 1
  3. 2025Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalizationcited by 1
  4. 2025LLM-Driven Kernel Evolution: Automating Driver Updates in Linuxcited by 1
  5. 2026BEAM: Bi-level Memory-adaptive Algorithmic Evolution for LLM-Powered Heuristic Designcited by 1
  6. 2026TraceDev: A Traceability-Driven Multi-agent Framework for Requirement-to-Code Developmentcited by 1
  7. 2026Simulation Code Generation for Fluid Systems using Large Language Models: Benchmarking Models and Prompting Strategiescited by 1
  8. 2026Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiencycited by 1
  9. 2025Do I look like a `cat.n.01` to you? A Taxonomy Image Generation Benchmarkcited by 0
  10. 2025LLMs in Coding and their Impact on the Commercial Software Engineering Landscapecited by 0
  11. 2025Testing chatbots on the creation of encoders for audio conditioned image generationcited by 0
  12. 2026Adaptive Confidence Gating in Multi-Agent Collaboration for Efficient and Optimized Code Generationcited by 0
  13. 2026COMPASS: The explainable agentic framework for Sovereignty, Sustainability, Compliance, and Ethicscited by 0
  14. 2026Characterizing the Quality Profile of AI-Generated C++ in Productioncited by 0