Understanding Bugs in Modern Agentic Frameworks: A Study of Symptoms, Root Causes, and Triggering Conditions
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Modern agentic frameworks such as CrewAI and AutoGen have evolved into complex, autonomous multi-agent systems, introducing reliability challenges that go beyond earlier pipeline-based LLM libraries. However, existing empirical studies focus on earlier LLM libraries or task-level bugs, leaving the unique complexities of these agentic frameworks unexplored. We present a comprehensive study of 409 fixed bugs across five representative agentic frameworks, proposing a five-layer architectural abstraction. Our taxonomy identifies previously unreported symptom categories---Unexpected Execution Sequence, User Configuration Ignored, and Incomplete/Incorrect Trace---and isolates agent-specific root causes including Model-Related Fault, Cognitive Context Mismanagement, and Orchestration Fault. Notably, the model integration layer is the most bug-prone yet receives disproportionately low test inclusion rate during bug fixing (47%), revealing a critical validation gap. Despite varying design paradigms, bug symptoms, root causes, and bug-prone components show substantial cross-framework consistency (JS similarity 0.62--0.88). Finally, we present the first systematic study of bug-triggering conditions, identifying error-prone factor combinations across element configurations, input patterns, and operations, and demonstrate their transferability across frameworks, providing a foundation for test oracle design and cross-framework benchmark.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 6 of its 51 references that are in the map. First posted on arXiv 10 Apr 2026.
Builds on
6 of 51 references.
Closest ancestors · 1
Same direction, newest first
Other references · 5
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2025Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Taskscited by 20
- 2025A First Look at Bugs in LLM Inference Enginescited by 5
- 2026A Large-Scale Study on the Development and Issues of Multi-Agent AI Systemscited by 5
- 2024Understanding and Detecting Annotation-Induced Faults of Static Analyzerscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reports
- 2026SelfHeal: Empirical Fix Pattern Analysis and Bug Repair in LLM Agents
- 2026Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes
- 2026Where Agent Frameworks Fall Short: Examining Functional Challenges and Usability Concerns
- 2026Automated structural testing of LLM-based agents: methods, framework, and case studies
- 2026MAESTRO: Multi-Agent Evaluation Suite for Testing, Reliability, and Observability
- 2026A Comprehensive Study of Implementation Bugs in Multi-modal Agents
- 2025Where LLM Agents Fail and How They can Learn From Failures