# Where Agent Frameworks Fall Short: Examining Functional Challenges and Usability Concerns arXiv: https://arxiv.org/abs/2602.21806 ยท HTML: https://mlsanity.com/paper/2602.21806/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 44 of its 66 references that are in the map. - First posted on arXiv 25 Feb 2026. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 30 of 44) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023) - [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023) - [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023) - [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023) - [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023) - [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023) - [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023) - [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023) - [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023) - [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023) - [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022) - [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024) - [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022) - [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024) - [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025) - [Identifying the Risks of LM Agents with an LM-Emulated Sandbox](https://mlsanity.com/paper/2309.15817/) (2023) - [AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents](https://mlsanity.com/paper/2401.13178/) (2024) - [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023) - [Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems](https://mlsanity.com/paper/2505.00212/) (2025) - [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024) - [TaskBench: Benchmarking Large Language Models for Task Automation](https://mlsanity.com/paper/2311.18760/) (2023) - [Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects](https://mlsanity.com/paper/2401.03428/) (2024) - [The Deep Learning Compiler: A Comprehensive Survey](https://mlsanity.com/paper/2002.03794/) (2020) - [NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers](https://mlsanity.com/paper/2207.13066/) (2022) - [Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Source](https://mlsanity.com/paper/2201.06589/) (2022) ## Built on by (most cited first, 3 of 3) - [Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reports](https://mlsanity.com/paper/2607.15684/) (2026) - [RepoRescue: An Empirical Study of LLM Agents on Whole-Repository Compatibility Rescue](https://mlsanity.com/paper/2607.01213/) (2026) - [Toward User-Conditioned Evaluation of Personal LLM Agents under Temporal Interventions](https://mlsanity.com/paper/2607.21635/) (2026) ## Abstract (arXiv) Large language model (LLM) agents are increasingly built on agent frameworks that provide reusable abstractions for workflow orchestration, state management, tool integration, and execution control. However, the quality of this infrastructure layer remains insufficiently understood, particularly its functionality challenges and usability concerns, as existing studies have mainly examined traditional deep learning (DL) frameworks or model-level agent failures. Therefore, we conduct an empirical study of 5,669 bug reports and 809 feature requests from five mainstream agent frameworks: AutoGen, CrewAI, LangChain, LangGraph, and MetaGPT. We construct a four-dimensional taxonomy covering 22 root causes, seven symptoms, 11 motivations, and six requirements, and map them to the five-stage agent lifecycle. Across the four RQs, results show an execution centered quality pattern shaped by semantic interface boundaries. Reported bugs mainly manifest as Incorrect Functionality (76.00%) and involve more API, configuration, parsing, and serialization related causes than DL framework bugs, while their associations remain sparse and stage specific. Feature requests mainly target Feature Enhancement (49.07%) and reveal structured needs for Orchestration Expressiveness, Development Delivery, Model Adaptation, and Tool Ecosystem. These findings call for quality assurance beyond crash based and tensor level testing, with emphasis on API sequences, structured LLM outputs, serialization boundaries, execution traces, and execution centered maintenance, offering empirical guidance for reliable and usable agent framework infrastructure.