mlsanityData to 14 Oct 2026

How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study

arXiv 2607.10856 ↗First posted 12 Jul 2026cs.SE cs.AI cs.HCcited by 2

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

The rise of Software Engineering (SE) agents, i.e., LLM-based agents that can understand large codebases and carry out engineering tasks with limited human intervention, has been marked by rapid advances and adoption, but little is known about how developers build these systems in practice: existing studies mine repositories or examine deployment, but few investigate how SE agents are constructed. Through semi-structured interviews with 20 practitioners from 12 organizations and an online survey of 80 practitioners, this paper is the first to study how SE processes are changing in the development of SE agents and what challenges developers face. We find that as implementation becomes cheaper, bottlenecks shift rather than disappear: long-standing work in requirements, coordination, and deployment becomes more visible, while reviewing generated code and evaluating agent behavior become new and increasingly central forms of work. We characterize a seven-stage workflow and five process shifts, including a move toward evaluation-driven development, in which evaluation is increasingly defined early and steers iteration, and the emergence of specifications as first-class artifacts that teams test and version alongside code. We further identify six challenges that teams face, together with 12 corresponding practices they use or propose to address them, including unreliable evaluation signals, comprehension debt as code outpaces understanding, and behavioral changes introduced by provider-side model updates.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 49 of its 96 references that are in the map. First posted on arXiv 12 Jul 2026.

Builds on

49 of 96 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions28 May
  2. 2026AgentSZZ: Teaching the LLM Agent to Play Detective with Bug-Inducing Commits3 Apr
  3. 2025An Empirical Study of Agent Developer Practices in AI Agent Frameworks1 Dec
  4. 2025What Challenges Do Developers Face in AI Agent Systems? An Empirical Study on Stack Overflow & GitHub Issues29 Oct
  5. 2025GA4GC: Greener Agent for Greener Code via Multi-Objective Configuration Optimization5 Oct
  6. 2025An Empirical Study of Testing Practices in Open Source AI Agent Frameworks and Agentic Applications23 Sep
  7. 2025A Survey on Code Generation with LLM-based Agents31 Jul
  8. 2025Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories23 Jun
  9. 2024AgentOps: Enabling Observability of LLM Agents8 Nov
  10. 2024Agents in Software Engineering: Survey, Landscape, and Vision13 Sep
  11. 2024Large Language Model-Based Agents for Software Engineering: A Survey4 Sep
  12. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future5 Aug
3 more
  1. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agents23 Jul
  2. 2024Agentless: Demystifying LLM-based Software Engineering Agents1 Jul
  3. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May

Other references · 34

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  4. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 538
  5. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 398
  6. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 279
  7. 2024AutoCodeRover: Autonomous Program Improvementcited by 278
  8. 2022Machine Learning Operations (MLOps): Overview, Definition, and Architecturecited by 128
  9. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 127
  10. 2025Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivitycited by 101
  11. 2023A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challengescited by 75
  12. 2021Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Processcited by 56
22 more
  1. 2025Measuring Agents in Productioncited by 37
  2. 2023A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitionerscited by 22
  3. 2025Professional Software Developers Don't Vibe, They Control: AI Agent Use for Coding in 2025cited by 20
  4. 2024An Empirical Study on Challenges for LLM Application Developerscited by 18
  5. 2023Building Your Own Product Copilot: Challenges, Opportunities, and Needscited by 17
  6. 2025"Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding Editorscited by 15
  7. 2023The Pipeline for the Continuous Development of Artificial Intelligence Models -- Current State of Research and Practicecited by 14
  8. 2025SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarioscited by 13
  9. 2024Beyond the Comfort Zone: Emerging Solutions to Overcome Challenges in Integrating LLMs into Software Productscited by 11
  10. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecturecited by 11
  11. 2023The Product Beyond the Model -- An Empirical Study of Repositories of Open-Source ML Productscited by 10
  12. 2023Evaluating SZZ Implementations: An Empirical Study on the Linux Kernelcited by 7
  13. 2026From Technical Debt to Cognitive and Intent Debt: Rethinking Software Health in the Age of AIcited by 6
  14. 2024Investigating the Role of Cultural Values in Adopting Large Language Models for Software Engineeringcited by 5
  15. 2025"My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistantscited by 4
  16. 2025Vibe Coding in Product Teams: Reconfiguring AI-Assisted Workflows, Prototyping, and Collaborationcited by 4
  17. 2018Knowledge Management in Software Engineering: A Systematic Review of Studied Concepts, Findings and Research Methods Usedcited by 3
  18. 2025Product Manager Practices for Delegating Work to Generative AI: "Accountability must not be delegated to non-human actors"cited by 3
  19. 2025The Fast and Spurious: Developer Productivity with GenAIcited by 3
  20. 2025Empirical and Sustainability Aspects of Software Engineering Research in the Era of Large Language Models: A Reflectioncited by 3
  21. 2025WhatsCode: Large-Scale GenAI Deployment for Developer Efficiency at WhatsAppcited by 2
  22. 2026LLM-Assisted Repository-Level Generation with Structured Spec-Driven Engineeringcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios
  2. 2026AgentSWE: Can Coding Agents Build the Agent You Actually Want?
  3. 2026The Semi-Executable Stack: Agentic Software Engineering and the Expanding Scope of SE
  4. 2026Rethinking Software Engineering for Agentic AI Systems
  5. 2025From Correctness to Collaboration: Toward a Human-Centered Framework for Evaluating AI Agent Behavior in Software Engineering
  6. 2026LLM-Based Agentic Systems for Software Engineering: Challenges and Opportunities
  7. 2025Agentic Software Engineering: Foundational Pillars and a Research Roadmap
  8. 2026Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents