# What Challenges Do Developers Face in AI Agent Systems? An Empirical Study on Stack Overflow & GitHub Issues

arXiv: https://arxiv.org/abs/2510.25423 · HTML: https://mlsanity.com/paper/2510.25423/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 76 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 23 of 23)

- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges](https://mlsanity.com/paper/2505.10468/) (2025)
- [AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways](https://mlsanity.com/paper/2406.02630/) (2024)
- [CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software](https://mlsanity.com/paper/2107.08760/) (2021)
- [Taxonomy of Real Faults in Deep Learning Systems](https://mlsanity.com/paper/1910.11015/) (2019)
- [Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents](https://mlsanity.com/paper/2505.02077/) (2025)
- [Generative to Agentic AI: Survey, Conceptualization, and Challenges](https://mlsanity.com/paper/2504.18875/) (2025)
- [Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition](https://mlsanity.com/paper/2507.20526/) (2025)
- [Building AI Agents for Autonomous Clouds: Challenges and Design Principles](https://mlsanity.com/paper/2407.12165/) (2024)
- [Improving Automated Code Reviews: Learning from Experience](https://mlsanity.com/paper/2402.03777/) (2024)
- [LimTopic: LLM-based Topic Modeling and Text Summarization for Analyzing Scientific Articles limitations](https://mlsanity.com/paper/2503.10658/) (2025)
- [We Need a New Ethics for a World of AI Agents](https://mlsanity.com/paper/2509.10289/) (2025)
- [Multilingual transformer and BERTopic for short text topic modeling: The case of Serbian](https://mlsanity.com/paper/2402.03067/) (2024)
- [Understanding Quantum Software Engineering Challenges An Empirical Study on Stack Exchange Forums and GitHub Issues](https://mlsanity.com/paper/2205.03181/) (2022)
- [AI Agent Communication from Internet Architecture Perspective: Challenges and Opportunities](https://mlsanity.com/paper/2509.02317/) (2025)
- [Challenges in Docker Development: A Large-scale Study Using Stack Overflow](https://mlsanity.com/paper/2008.04467/) (2020)
- [An Empirical Study of ChatGPT-Related Projects and Their Issues on GitHub](https://mlsanity.com/paper/2403.17437/) (2024)
- [Mining Architecture Tactics and Quality Attributes Knowledge in Stack Overflow](https://mlsanity.com/paper/2105.07940/) (2021)
- [Unveiling Ruby: Insights from Stack Overflow and Developer Survey](https://mlsanity.com/paper/2503.19238/) (2025)
- [Exploring Challenges in Test Mocking: Developer Questions and Insights from StackOverflow](https://mlsanity.com/paper/2505.08300/) (2025)
- [Developer Discussion Topics on the Adoption and Barriers of Low Code Software Development Platforms](https://mlsanity.com/paper/2209.00844/) (2022)
- [Energy-Efficient Software Development: A Multi-dimensional Empirical Analysis of Stack Overflow](https://mlsanity.com/paper/2409.19222/) (2024)

## Built on by (most cited first, 2 of 2)

- [How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study](https://mlsanity.com/paper/2607.10856/) (2026)
- [Understanding Issues, Causes and Solutions in Open-Source LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2610.00905/) (2026)

## Abstract (arXiv)

AI Agents have rapidly gained prominence in both research and industry as systems that extend large language models with planning, tool use, memory, and goal-directed action. Despite this progress, the development and maintenance of Agent systems present recurring engineering difficulties that are not yet well characterized in developer-facing evidence. To address this gap, this study analyzes developer discussions on Stack Overflow and failure reports from GitHub issue trackers associated with widely used Agent frameworks. For Stack Overflow, an Agent-focused corpus is constructed through tag expansion and filtering, latent themes are derived using LDA-MALLET, and topics are manually validated and labeled. For GitHub, a taxonomy of issue themes is developed to capture deployment-time failures and maintenance burdens. Analysis across both platforms identifies seven Stack Overflow topics (comprising 28 subtopics) and thirteen GitHub issue topics, which are synthesized into five overarching families of major Agent challenges: (1) environment, platforms, and dependency management; (2) retrieval, embeddings, and Agent memory; (3) orchestration and execution control; (4) interaction contracts between models and tools; and (5) runtime reliability and operational robustness. Topic popularity and difficulty are quantified, revealing that widely discussed issues, such as installation and prompting, are often resolved more quickly, whereas retrieval- and orchestration-related challenges are less visible, more complex, and tend to persist as ongoing maintenance burdens on GitHub.