# An Empirical Study of Testing Practices in Open Source AI Agent Frameworks and Agentic Applications arXiv: https://arxiv.org/abs/2509.19185 ยท HTML: https://mlsanity.com/paper/2509.19185/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 11 of its 119 references that are in the map. - First posted on arXiv 23 Sep 2025. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 11 of 11) - [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023) - [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023) - [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023) - [The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey](https://mlsanity.com/paper/2404.11584/) (2024) - [Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects](https://mlsanity.com/paper/2401.03428/) (2024) - [Model Context Protocol (MCP) at First Glance: Studying the Security and Maintainability of MCP Servers](https://mlsanity.com/paper/2506.13538/) (2025) - [Challenges in Testing Large Language Model Based Software: A Faceted Taxonomy](https://mlsanity.com/paper/2503.00481/) (2025) - [Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol](https://mlsanity.com/paper/2508.20737/) (2025) - [Prompt Baking](https://mlsanity.com/paper/2409.13697/) (2024) - [Test & Evaluation Best Practices for Machine Learning-Enabled Systems](https://mlsanity.com/paper/2310.06800/) (2023) - [From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmap](https://mlsanity.com/paper/2410.20791/) (2024) ## Built on by (most cited first, 15 of 15) - [Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions](https://mlsanity.com/paper/2602.14878/) (2026) - [An Empirical Study of Agent Developer Practices in AI Agent Frameworks](https://mlsanity.com/paper/2512.01939/) (2025) - [Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes](https://mlsanity.com/paper/2603.06847/) (2026) - [Beyond Task Completion: An Assessment Framework for Evaluating Agentic AI Systems](https://mlsanity.com/paper/2512.12791/) (2025) - [How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests](https://mlsanity.com/paper/2601.17581/) (2026) - [Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub](https://mlsanity.com/paper/2601.00477/) (2026) - [Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality](https://mlsanity.com/paper/2607.03691/) (2026) - [Toward Architecture-Aware Evaluation Metrics for LLM Agents](https://mlsanity.com/paper/2601.19583/) (2026) - [ADK Arena: Evaluating Agent Development Kits via LLM-as-a-Developer](https://mlsanity.com/paper/2606.05548/) (2026) - [How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study](https://mlsanity.com/paper/2607.10856/) (2026) - [Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications](https://mlsanity.com/paper/2608.08413/) (2026) - [Agentic Frameworks for Reasoning Tasks: An Empirical Study](https://mlsanity.com/paper/2604.16646/) (2026) - [A Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI Agents](https://mlsanity.com/paper/2609.17698/) (2026) - [AgentBug-Smith: Automatically Reproducing Real-World Harness Bugs in Agentic Systems](https://mlsanity.com/paper/2609.37864/) (2026) - [Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild](https://mlsanity.com/paper/2610.04921/) (2026) ## Abstract (arXiv) Foundation model (FM)-based AI agents are rapidly gaining adoption across diverse domains, but their inherent non-determinism and non-reproducibility pose testing and quality assurance challenges. While recent benchmarks provide task-level evaluations, there is limited understanding of how developers verify the internal correctness of these agents during development. To address this gap, we conduct the first large-scale empirical study of testing practices in the AI agent ecosystem, analyzing 39 open-source agent frameworks and 439 agentic applications. We identify ten distinct testing patterns and find that novel, agent-specific methods like DeepEval are seldom used (around 1%), while traditional patterns like negative and membership testing are widely adapted to manage FM uncertainty. By mapping these patterns to canonical architectural components of agent frameworks and agentic applications, we uncover a fundamental inversion of testing effort: deterministic components like Resource Artifacts (tools) and Coordination Artifacts (workflows) consume over 70% of testing effort, while the FM-based Plan Body receives less than 5%. Crucially, this reveals a critical blind spot, as the Trigger component (prompts) remains neglected, appearing in around 1% of all tests. Our findings offer the first empirical testing baseline in FM-based agent frameworks and agentic applications, revealing a rational but incomplete adaptation to non-determinism. To address it, framework developers should improve support for novel testing methods, application developers must adopt prompt regression testing, and researchers should explore barriers to adoption. Strengthening these practices is vital for building more robust and dependable AI agents.