mlsanityData to 7 Oct 2026

An Empirical Study of Agent Developer Practices in AI Agent Frameworks

arXiv 2512.01939 ↗First posted 1 Dec 2025cs.SE cs.AIcited by 14

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

The rise of large language models (LLMs) has sparked a surge of interest in agents, leading to the rapid growth of agent frameworks. Agent frameworks are software toolkits and libraries that provide standardized components, abstractions, and orchestration mechanisms to simplify agent development. Despite widespread use of agent frameworks, their practical applications and how they influence the agent development process remain underexplored. Different agent frameworks encounter similar problems during use, indicating that these recurring issues deserve greater attention and call for further improvements in agent framework design. Meanwhile, as the number of agent frameworks continues to grow and evolve, more than 80% of developers report difficulties in identifying the frameworks that best meet their specific development requirements. In this paper, we conduct the first empirical study of LLM-based agent frameworks, exploring real-world experiences of developers in building AI agents. To compare how well the agent frameworks meet developer needs, we further collect developer discussions for the ten previously identified agent frameworks, resulting in a total of 11,910 discussions. Finally, by analyzing these discussions, we compare the frameworks across five dimensions: development efficiency, functional abstraction, learning cost, performance optimization, and maintainability, which refers to how easily developers can update and extend both the framework itself and the agents built upon it over time. Our comparative analysis reveals significant differences among frameworks in how they meet the needs of agent developers. Overall, we provide a set of findings and implications for the LLM-driven AI agent framework ecosystem and offer insights for the design of future LLM-based agent frameworks and agent developers.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 59 of its 105 references that are in the map. First posted on arXiv 1 Dec 2025.

Builds on

59 of 105 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025An Empirical Study of Testing Practices in Open Source AI Agent Frameworks and Agentic Applications23 Sep
  2. 2025Why Do Multi-Agent LLM Systems Fail?17 Mar
  3. 2024Human-In-the-Loop Software Development Agents19 Nov
  4. 2024Agents in Software Engineering: Survey, Landscape, and Vision13 Sep
  5. 2024Agentless: Demystifying LLM-based Software Engineering Agents1 Jul
  6. 2024CoSQA+: Pioneering the Multi-Choice Code Search Benchmark with Test-Driven Agents17 Jun
  7. 2024MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution26 Mar
  8. 2024CodeAgent: Autonomous Communicative Agents for Code Review3 Feb
  9. 2023Autonomous Agents in Software Development: A Vision Paper30 Nov

Other references · 50

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  5. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  6. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  7. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  8. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  9. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  10. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  11. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  12. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
38 more
  1. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  2. 2024AI Agents That Mattercited by 199
  3. 2025Survey on Evaluation of LLM-based Agentscited by 190
  4. 2024The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Surveycited by 164
  5. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 140
  6. 2024The Prompt Report: A Systematic Survey of Prompt Engineering Techniquescited by 115
  7. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  8. 2025A survey of agent interoperability protocols: Model Context Protocol (MCP), Agent Communication Protocol (ACP), Agent-to-Agent Protocol (A2A), and Agent Network Protocol (ANP)cited by 98
  9. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 92
  10. 2024CodeR: Issue Resolving with Multi-Agent and Task Graphscited by 89
  11. 2023Towards an Understanding of Large Language Models in Software Engineering Taskscited by 67
  12. 2024Agent Planning with World Knowledge Modelcited by 66
  13. 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 49
  14. 2021On the Evaluation of Neural Code Summarizationcited by 47
  15. 2023When ChatGPT Meets Smart Contract Vulnerability Detection: How Far Are We?cited by 42
  16. 2024SWE-bench-java: A GitHub Issue Resolving Benchmark for Javacited by 34
  17. 2023Agent Instructs Large Language Models to be General Zero-Shot Reasonerscited by 32
  18. 2024RMCBench: Benchmarking Large Language Models' Resistance to Malicious Codecited by 27
  19. 2024HumanEvo: An Evolution-aware Benchmark for More Realistic Evaluation of Repository-level Code Generationcited by 26
  20. 2025OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue Resolutioncited by 24
  21. 2025What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyondcited by 22
  22. 2022CoCoSoDa: Effective Contrastive Learning for Code Searchcited by 21
  23. 2024STALL+: Boosting LLM-based Repository-level Code Completion with Static Analysiscited by 21
  24. 2024When to Stop? Towards Efficient Code Generation in LLMs with Excess Token Preventioncited by 21
  25. 2024RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translationcited by 21
  26. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replaycited by 19
  27. 2024ACEGEN: Reinforcement learning of generative chemical agents for drug discoverycited by 14
  28. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
  29. 2024SparseCoder: Identifier-Aware Sparse Transformer for File-Level Code Summarizationcited by 11
  30. 2024You Augment Me: Exploring ChatGPT-based Data Augmentation for Semantic Code Searchcited by 11
  31. 2025EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solvingcited by 10
  32. 2022Tackling Long Code Search with Splitting, Encoding, and Aggregatingcited by 9
  33. 2025RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generationcited by 9
  34. 2024Identifying Smart Contract Security Issues in Code Snippets from Stack Overflowcited by 8
  35. 2021Learning Interaction Kernels for Agent Systems on Riemannian Manifoldscited by 7
  36. 2024Aegis:An Advanced LLM-Based Multi-Agent for Intelligent Functional Safety Engineeringcited by 7
  37. 2023Improving Code Search with Hard Negative Sampling Based on Fine-tuningcited by 5
  38. 2025Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 13.

14 papers.

Sort:

  1. 2025An Information Theoretic Perspective on Agentic System Designcited by 8
  2. 2026From Failure to Mastery: Generating Hard Samples for Tool-use Agentscited by 8
  3. 2026AI-NativeBench: An Open-Source White-Box Agentic Benchmark Suite for AI-Native Systemscited by 3
  4. 2026Where Agent Frameworks Fall Short: Examining Functional Challenges and Usability Concernscited by 3
  5. 2026In-Context Prompting Obsoletes Agent Orchestration for Procedural Taskscited by 2
  6. 2026ADK Arena: Evaluating Agent Development Kits via LLM-as-a-Developercited by 2
  7. 2026The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behaviorcited by 2
  8. 2026Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelinescited by 1
  9. 2026How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Studycited by 1
  10. 2026Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applicationscited by 1
  11. 2026Declarative by Design, Assistable Only by Convention: Benchmarking Multi-Agent Frameworks for AI-Assistabilitycited by 0
  12. 2026Distributed Legal Infrastructure for a Trustworthy Agentic Webcited by 0
2 more
  1. 2026Agentic Frameworks for Reasoning Tasks: An Empirical Studycited by 0
  2. 2026AgentFlow: Building Agent Dependency Graphs for Static Analysis of Agent Programscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report
  2. 2025Efficient Agents: Building Effective Agents While Reducing Cost
  3. 2024CodePori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
  4. 2026AgentFactory: Towards Automated Agentic System Design and Optimization
  5. 2024Large Language Model-Based Agents for Software Engineering: A Survey
  6. 2025AgentSwift: Efficient LLM Agent Design via Value-guided Hierarchical Search
  7. 2026Agentic AI in the Software Development Lifecycle: Architecture, Empirical Evidence, and the Reshaping of Software Engineering
  8. 2025OAgents: An Empirical Study of Building Effective Agents