mlsanityData to 7 Oct 2026

Large Language Models for Software Engineering: Survey and Open Problems

arXiv 2310.03533 ↗First posted 5 Oct 2023cs.SEcited by 178

Not on the map. Why?

Cited by 38 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper provides a survey of the emerging area of Large Language Models (LLMs) for Software Engineering (SE). It also sets out open research challenges for the application of LLMs to technical problems faced by software engineers. LLMs' emergent properties bring novelty and creativity with applications right across the spectrum of Software Engineering activities including coding, design, requirements, repair, refactoring, performance improvement, documentation and analytics. However, these very same emergent properties also pose significant technical challenges; we need techniques that can reliably weed out incorrect solutions, such as hallucinations. Our survey reveals the pivotal role that hybrid techniques (traditional SE plus LLMs) have to play in the development and deployment of reliable, efficient and effective LLM-based SE.

Key facts. Cited by 178 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Oct 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 60; 2025: 60; 2026: 55.

178 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  2. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  3. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  4. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  5. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
  6. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 89
  7. 2024RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graphcited by 88
  8. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
  9. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 62
  10. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  11. 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarioscited by 49
  12. 2024Reasoning Runtime Behavior of a Program with LLM: How Far Are We?cited by 44
38 more
  1. 2024Are LLMs Ready for Real-World Materials Discovery?cited by 40
  2. 2025EnvBench: A Benchmark for Automated Environment Setupcited by 36
  3. 2024Source Code Summarization in the Era of Large Language Modelscited by 30
  4. 2024EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimizationcited by 28
  5. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 26
  6. 2024HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agentcited by 24
  7. 2024The Future of AI-Driven Software Engineeringcited by 23
  8. 2025Mutation-Guided LLM-based Test Generation at Metacited by 23
  9. 2025OJBench: A Competition Level Code Benchmark For Large Language Modelscited by 22
  10. 2024Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projectscited by 21
  11. 2024Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysiscited by 20
  12. 2024Large Language Models as Software Components: A Taxonomy for LLM-Integrated Applicationscited by 17
  13. 2025Correctness Assessment of Code Generated by Large Language Models Using Internal Representationscited by 17
  14. 2025Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agentscited by 17
  15. 2024The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Modelscited by 15
  16. 2024InterTrans: Leveraging Transitive Intermediate Translations to Enhance LLM-based Code Translationcited by 15
  17. 2025Generative AI for Software Architecture. Applications, Challenges, and Future Directionscited by 15
  18. 2025Large Language Models (LLMs) for Source Code Analysis: applications, models and datasetscited by 15
  19. 2025Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Modelscited by 14
  20. 2024InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Modelscited by 13
  21. 2024AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Accelerationcited by 13
  22. 2025MaintainCoder: Maintainable Code Generation Under Dynamic Requirementscited by 13
  23. 2025TAMO: Fine-Grained Root Cause Analysis via Tool-Assisted LLM Agent with Multi-Modality Observation Data in Cloud-Native Systemscited by 13
  24. 2026Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understandingcited by 13
  25. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM modelscited by 12
  26. 2025Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studiescited by 12
  27. 2024Model Generation with LLMs: From Requirements to UML Sequence Diagramscited by 11
  28. 2024Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Testscited by 11
  29. 2026Wink: Recovering from Misbehaviors in Coding Agentscited by 11
  30. 2024Struct-X: Enhancing Large Language Models Reasoning with Structured Datacited by 10
  31. 2024RAMBO: Enhancing RAG-based Repository-Level Method Body Completioncited by 10
  32. 2025Agentic AI Process Observability: Discovering Behavioral Variabilitycited by 10
  33. 2023Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Codecited by 9
  34. 2024Output Format Biases in the Evaluation of Large Language Models for Code Translationcited by 9
  35. 2024Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Modelscited by 9
  36. 2024Natural Language Outlines for Code: Literate Programming in the LLM Eracited by 9
  37. 2024A Survey on Symbolic Knowledge Distillation of Large Language Modelscited by 9
  38. 2024On Large Language Models in Mission-Critical IT Governance: Are We Ready Yet?cited by 9

50 most cited of 178 All 178 (JSON)