mlsanityData to 7 Oct 2026

Large Language Models for Software Engineering: A Systematic Literature Review

arXiv 2308.10620 ↗First posted 21 Aug 2023cs.SE cs.AIcited by 389

Not on the map. Why?

Cited by 101 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) have significantly impacted numerous domains, including Software Engineering (SE). Many recent publications have explored LLMs applied to various SE tasks. Nevertheless, a comprehensive understanding of the application, effects, and possible limitations of LLMs on SE is still in its early stages. To bridge this gap, we conducted a systematic literature review (SLR) on LLM4SE, with a particular focus on understanding how LLMs can be exploited to optimize processes and outcomes. We select and analyze 395 research papers from January 2017 to January 2024 to answer four key research questions (RQs). In RQ1, we categorize different LLMs that have been employed in SE tasks, characterizing their distinctive features and uses. In RQ2, we analyze the methods used in data collection, preprocessing, and application, highlighting the role of well-curated datasets for successful LLM for SE implementation. RQ3 investigates the strategies employed to optimize and evaluate the performance of LLMs in SE. Finally, RQ4 examines the specific SE tasks where LLMs have shown success to date, illustrating their practical contributions to the field. From the answers to these RQs, we discuss the current state-of-the-art and trends, identifying gaps in existing research, and flagging promising areas for future study. Our artifacts are publicly available at https://github.com/xinyi-hou/LLM4SE_SLR.

Key facts. Cited by 389 papers in the map (data through 7 Oct 2026). Builds on 256 of its 408 references that are in the map. First posted on arXiv 21 Aug 2023.

Builds on

256 of 408 references.

References · 256

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  11. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  12. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  13. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  14. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  15. 2023Teaching Large Language Models to Self-Debugcited by 963
  16. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  17. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  18. 2021Measuring Coding Challenge Competence With APPScited by 879
  19. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  20. 2015Gated Graph Sequence Neural Networkscited by 792
  21. 2022Galactica: A Large Language Model for Sciencecited by 736
  22. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  23. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  24. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  25. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  26. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  27. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  28. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  29. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  30. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  31. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  32. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  33. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  34. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  35. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  36. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  37. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  38. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346

50 most cited of 256 All 256 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 100; 2025: 155; 2026: 120.

389 papers.

Sort:

By area:

  1. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  2. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  3. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
  4. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 133
  5. 2024MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolutioncited by 132
  6. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  7. 2024RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graphcited by 88
  8. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
  9. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  10. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 65
  11. 2024A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languagescited by 65
  12. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
38 more
  1. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  2. 2024CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledgecited by 55
  3. 2025On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHubcited by 53
  4. 2023Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectivescited by 48
  5. 2024Long Code Arena: a Set of Benchmarks for Long-Context Code Modelscited by 48
  6. 2024Reasoning Runtime Behavior of a Program with LLM: How Far Are We?cited by 44
  7. 2024A Scalable Communication Protocol for Networks of Large Language Modelscited by 43
  8. 2024Human-In-the-Loop Software Development Agentscited by 42
  9. 2024Meta Large Language Model Compiler: Foundation Models of Compiler Optimizationcited by 39
  10. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 39
  11. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  12. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 34
  13. 2024Data-driven Discovery with Large Generative Modelscited by 33
  14. 2024Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuningcited by 31
  15. 2024Source Code Summarization in the Era of Large Language Modelscited by 30
  16. 2024Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmapcited by 30
  17. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 29
  18. 2024The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmerscited by 28
  19. 2024MARE: Multi-Agents Collaboration Framework for Requirements Engineeringcited by 28
  20. 2025Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystemcited by 28
  21. 2023LLM-Assisted Code Cleaning For Training Accurate Code Generatorscited by 27
  22. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 26
  23. 2024Agents4PLC: Automating Closed-loop PLC Code Generation and Verification in Industrial Control Systems using LLM-based Agentscited by 25
  24. 2024How secure is AI-generated Code: A Large-Scale Comparison of Large Language Modelscited by 23
  25. 2025Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projectscited by 23
  26. 2024CodePori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technologycited by 21
  27. 2024Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projectscited by 21
  28. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  29. 2024Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysiscited by 20
  30. 2024ControlAgent: Automating Control System Design via Novel Integration of LLM Agents and Domain Expertisecited by 20
  31. 2023Trustworthy and Synergistic Artificial Intelligence for Software Engineering: Vision and Roadmapscited by 19
  32. 2024An overview of domain-specific foundation model: key technologies, applications and challengescited by 19
  33. 2024Large Language Models as Software Components: A Taxonomy for LLM-Integrated Applicationscited by 17
  34. 2025Correctness Assessment of Code Generated by Large Language Models Using Internal Representationscited by 17
  35. 2024AcTracer: Active Testing of Large Language Model via Multi-Stage Samplingcited by 16
  36. 2024Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarioscited by 16
  37. 2025Large Language Models for Computer-Aided Design: A Surveycited by 16
  38. 2024Code Simulation Challenges for Large Language Modelscited by 15

50 most cited of 389 All 389 (JSON)