mlsanityData to 7 Oct 2026

Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios

arXiv 2410.12468 ↗First posted 16 Oct 2024cs.SE cs.AIcited by 16

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

In recent years, AI-based software engineering has progressed from pre-trained models to advanced agentic workflows, with Software Development Agents representing the next major leap. These agents, capable of reasoning, planning, and interacting with external environments, offer promising solutions to complex software engineering tasks. However, while much research has evaluated code generated by large language models (LLMs), comprehensive studies on agent-generated patches, particularly in real-world settings, are lacking. This study addresses that gap by evaluating 4,892 patches from 10 top-ranked agents on 500 real-world GitHub issues from SWE-Bench Verified, focusing on their impact on code quality. Our analysis shows no single agent dominated, with 170 issues unresolved, indicating room for improvement. Even for patches that passed unit tests and resolved issues, agents made different file and function modifications compared to the gold patches from repository developers, revealing limitations in the benchmark's test case coverage. Most agents maintained code reliability and security, avoiding new bugs or vulnerabilities; while some agents increased code complexity, many reduced code duplication and minimized code smells. Finally, agents performed better on simpler codebases, suggesting that breaking complex tasks into smaller sub-tasks could improve effectiveness. This study provides the first comprehensive evaluation of agent-generated patches on real-world GitHub issues, offering insights to advance AI-driven software development.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 52 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

24 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023ChatDev: Communicative Agents for Software Development16 Jul

Other references · 23

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  4. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  5. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  6. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  7. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 149
  8. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 136
  9. 2023Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issuescited by 78
  10. 2022Automated Repair of Programs from Large Language Modelscited by 71
  11. 2023Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learningcited by 71
  12. 2022Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?cited by 63
11 more
  1. 2023LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluationscited by 58
  2. 2023No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPTcited by 45
  3. 2023Assessing the Security of GitHub Copilot Generated Code -- A Targeted Replication Studycited by 24
  4. 2023On the Reliability and Explainability of Language Models for Program Generationcited by 17
  5. 2024LLM Security Guard for Codecited by 15
  6. 2023A User-centered Security Evaluation of Copilotcited by 14
  7. 2024Just another copy and paste? Comparing the security vulnerabilities of ChatGPT generated code and StackOverflow answerscited by 12
  8. 2024Promise and Peril of Collaborative Code Generation Models: Balancing Effectiveness and Memorizationcited by 4
  9. 2023Delving into Commit-Issue Correlation to Enhance Commit Message Generation Modelscited by 2
  10. 2018On Relating Technical, Social Factors, and the Introduction of Bugscited by 1
  11. 2021Fixing Vulnerabilities Potentially Hinders Maintainabilitycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 10.

16 papers.

Sort:

By area:

  1. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  2. 2025UTBoost: Rigorous Evaluation of Coding Agents on SWE-Benchcited by 44
  3. 2026SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Taskscited by 24
  4. 2025Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Studycited by 16
  5. 2026Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understandingcited by 13
  6. 2026Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboardscited by 12
  7. 2026How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requestscited by 7
  8. 2025Agentic Software Issue Resolution with Large Language Models: A Surveycited by 6
  9. 2026TraceCoder: A Trace-Driven Multi-Agent Framework for Automated Debugging of LLM-Generated Codecited by 6
  10. 2025TDFlow: Agentic Workflows for Test Driven Developmentcited by 5
  11. 2025CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournamentscited by 5
  12. 2026SWE-Prime: Fewer Trajectories, Better Performancecited by 3
4 more
  1. 2026"Refactoring Runaway": Understanding and Mitigating Tangled Refactorings in Coding Agents for Issue Resolutioncited by 1
  2. 2026Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?cited by 1
  3. 2026Do Influence Tactics Matter? Investigating Prompt Framing Effects in LLM Code Generationcited by 0
  4. 2026Engineering Signals of Human-AI Collaboration in the Agentic Coding Era: A Longitudinal Analysis of 33,228 Pull Requests from vLLM and SGLang with Implications for Biomedical AI Agents and Bioinformatics Pipeline Developmencited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
  2. 2025Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios
  3. 2026Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub
  4. 2026Beyond Bug Fixes: An Empirical Investigation of Post-Merge Code Quality Issues in Agent-Generated Pull Requests
  5. 2026Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Time
  6. 2025Agentic Refactoring: An Empirical Study of AI Coding Agents
  7. 2026AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub
  8. 2026AI builds, We Analyze: An Empirical Study of AI-Generated Build Code Quality