# A Survey on Large Language Models for Software Engineering

arXiv: https://arxiv.org/abs/2312.15223 · HTML: https://mlsanity.com/paper/2312.15223/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2023.

## Built on by (most cited first, 30 of 59)

- [SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios](https://mlsanity.com/paper/2512.18470/) (2025)
- [Source Code Summarization in the Era of Large Language Models](https://mlsanity.com/paper/2407.07959/) (2024)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures](https://mlsanity.com/paper/2604.03515/) (2026)
- [Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing](https://mlsanity.com/paper/2506.16136/) (2025)
- [ReCode: Updating Code API Knowledge with Reinforcement Learning](https://mlsanity.com/paper/2506.20495/) (2025)
- [Conversational AI as a Coding Assistant: Understanding Programmers' Interactions with and Expectations from Large Language Models for Coding](https://mlsanity.com/paper/2503.16508/) (2025)
- [SGAgent: Suggestion-Guided LLM-Based Multi-Agent Framework for Repository-Level Software Repair](https://mlsanity.com/paper/2602.23647/) (2026)
- [ESALE: Enhancing Code-Summary Alignment Learning for Source Code Summarization](https://mlsanity.com/paper/2407.01646/) (2024)
- [Can GPT-O1 Kill All Bugs? An Evaluation of GPT-Family LLMs on QuixBugs](https://mlsanity.com/paper/2409.10033/) (2024)
- [MAVUL: Multi-Agent Vulnerability Detection via Contextual Reasoning and Interactive Refinement](https://mlsanity.com/paper/2510.00317/) (2025)
- [Defending Code Language Models against Backdoor Attacks with Deceptive Cross-Entropy Loss](https://mlsanity.com/paper/2407.08956/) (2024)
- [The Impact of Hyperparameters on Large Language Model Inference Performance: An Evaluation of vLLM and HuggingFace Pipelines](https://mlsanity.com/paper/2408.01050/) (2024)
- [RobuNFR: Evaluating the Robustness of Large Language Models on Non-Functional Requirements Aware Code Generation](https://mlsanity.com/paper/2503.22851/) (2025)
- [YABLoCo: Yet Another Benchmark for Long Context Code Generation](https://mlsanity.com/paper/2505.04406/) (2025)
- [Text-to-Pipeline: Bridging Natural Language and Data Preparation Pipelines](https://mlsanity.com/paper/2505.15874/) (2025)
- [AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context](https://mlsanity.com/paper/2601.19494/) (2026)
- [Specification-Based Code-Text-Code Reengineering for LLM-Mediated Software Evolution](https://mlsanity.com/paper/2605.25232/) (2026)
- [Improving Deep Assertion Generation via Fine-Tuning Retrieval-Augmented Pre-trained Language Models](https://mlsanity.com/paper/2502.16071/) (2025)
- [Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions](https://mlsanity.com/paper/2507.21285/) (2025)
- [The Future of Generative AI in Software Engineering: A Vision from Industry and Academia in the European GENIUS Project](https://mlsanity.com/paper/2511.01348/) (2025)
- [Dynamic Stability of LLM-Generated Code](https://mlsanity.com/paper/2511.07463/) (2025)
- [Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering](https://mlsanity.com/paper/2604.16790/) (2026)
- [Mastering the Craft of Data Synthesis for CodeLLMs](https://mlsanity.com/paper/2411.00005/) (2024)
- [Novice Developers' Perspectives on Adopting LLMs for Software Development: A Systematic Literature Review](https://mlsanity.com/paper/2503.07556/) (2025)
- [Domain Specific Benchmarks for Evaluating Multimodal Large Language Models](https://mlsanity.com/paper/2506.12958/) (2025)
- [Do AI Coding Agents Log Like Humans? An Empirical Study](https://mlsanity.com/paper/2604.09409/) (2026)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)
- [Exploring the Integration of Large Language Models in Industrial Test Maintenance Processes](https://mlsanity.com/paper/2409.06416/) (2024)
- [Improving Retrieval-Augmented Deep Assertion Generation via Joint Training](https://mlsanity.com/paper/2502.10696/) (2025)

## Abstract (arXiv)

Software Engineering (SE) is the systematic design, development, maintenance, and management of software applications underpinning the digital infrastructure of our modern world. Very recently, the SE community has seen a rapidly increasing number of techniques employing Large Language Models (LLMs) to automate a broad range of SE tasks. Nevertheless, existing information of the applications, effects, and possible limitations of LLMs within SE is still not well-studied. In this paper, we provide a systematic survey to summarize the current state-of-the-art research in the LLM-based SE community. We summarize 62 representative LLMs of Code across three model architectures, 15 pre-training objectives across four categories, and 16 downstream tasks across five categories. We then present a detailed summarization of the recent SE studies for which LLMs are commonly utilized, including 947 studies for 112 specific code-related tasks across five crucial phases within the SE workflow. We also discuss several critical aspects during the integration of LLMs into SE, such as empirical evaluation, benchmarking, security and reliability, domain tuning, compressing and distillation. Finally, we highlight several challenges and potential opportunities on applying LLMs for future SE studies, such as exploring domain LLMs and constructing clean evaluation datasets. Overall, our work can help researchers gain a comprehensive understanding about the achievements of the existing LLM-based SE studies and promote the practical application of these techniques. Our artifacts are publicly available and will be continuously updated at the living repository: https://github.com/iSEngLab/AwesomeLLM4SE.