# Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications arXiv: https://arxiv.org/abs/2608.08413 · HTML: https://mlsanity.com/paper/2608.08413/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 31 of its 90 references that are in the map. - First posted on arXiv 9 Aug 2026. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 30 of 31) - [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023) - [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023) - [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023) - [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024) - [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023) - [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024) - [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025) - [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](https://mlsanity.com/paper/2509.16941/) (2025) - [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024) - [MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents](https://mlsanity.com/paper/2503.01935/) (2025) - [SWE-Bench+: Enhanced Coding Benchmark for LLMs](https://mlsanity.com/paper/2410.06992/) (2024) - [Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories](https://mlsanity.com/paper/2506.18824/) (2025) - [Interactive Debugging and Steering of Multi-Agent AI Systems](https://mlsanity.com/paper/2503.02068/) (2025) - [AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents](https://mlsanity.com/paper/2505.05849/) (2025) - [ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions](https://mlsanity.com/paper/2601.06112/) (2026) - [User Modeling and User Profiling: A Comprehensive Survey](https://mlsanity.com/paper/2402.09660/) (2024) - [What's in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform](https://mlsanity.com/paper/1811.07643/) (2018) - [Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents](https://mlsanity.com/paper/2510.02837/) (2025) - [Chaos Engineering](https://mlsanity.com/paper/1702.05843/) (2017) - [Agentic AI Needs a Systems Theory](https://mlsanity.com/paper/2503.00237/) (2025) - [An Empirical Study of Testing Practices in Open Source AI Agent Frameworks and Agentic Applications](https://mlsanity.com/paper/2509.19185/) (2025) - [An Empirical Study of Agent Developer Practices in AI Agent Frameworks](https://mlsanity.com/paper/2512.01939/) (2025) - [Scalpel: The Python Static Analysis Framework](https://mlsanity.com/paper/2202.11840/) (2022) - [ChainFuzzer: Greybox Fuzzing for Workflow-Level Multi-Tool Vulnerabilities in LLM Agents](https://mlsanity.com/paper/2603.12614/) (2026) - [Challenges in Testing Large Language Model Based Software: A Faceted Taxonomy](https://mlsanity.com/paper/2503.00481/) (2025) - [Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem](https://mlsanity.com/paper/2510.09907/) (2025) - [WebGraphEval: Multi-Turn Trajectory Evaluation for Web Agents using Graph Representation](https://mlsanity.com/paper/2510.19205/) (2025) - [A survey on software testability](https://mlsanity.com/paper/1801.02201/) (2018) - [Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol](https://mlsanity.com/paper/2508.20737/) (2025) - [A Control-Theoretic Foundation for Agentic Systems](https://mlsanity.com/paper/2603.10779/) (2026) ## Built on by (most cited first, 2 of 2) - [Towards Risk-free AI Agent Deployment](https://mlsanity.com/paper/2608.16411/) (2026) - [Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild](https://mlsanity.com/paper/2610.04921/) (2026) ## Abstract (arXiv) Agents built on large language models (LLMs) are increasingly used to build applications that perform complex, multi-step tasks involving reasoning, tool use, and interaction with external environments. Despite rapid progress in benchmarking LLM-based agents, very few studies have attempted to understand how such systems are tested in practice. In particular, testing levels, objectives, data patterns, test complexity, and validation strategies for agent applications remain underexplored. In this paper, we present an empirical study of testing practices in LLM-based agent applications using a large corpus of mined open-source projects. We construct a large-scale dataset of agent applications, tools, and tests, and manually label 2,572 test methods from 240 modules. From this analysis, we derive a taxonomy of 23 testing patterns across test fixtures, data, objectives, and assertions, and characterize tests by level (unit, module, integration). We complement this with structured interviews of 10 senior industry practitioners building agentic systems. Our results show that testing of LLM-based agent applications is dominated by narrowly scoped unit tests, with limited coverage of complex interactions, realistic scenarios, and non-functional requirements. Tests frequently rely on simplistic inputs, heavy mocking, and shallow validation, and agent-related tests exhibit low structural complexity. While industry practice places greater emphasis on non-functional testing than open-source projects, both reveal common gaps, including the lack of formal testing foundations, unclear test objectives, and challenges in generating high-quality test data. Based on these findings, we outline research directions toward more systematic and rigorous testing of agent applications, including foundations for agent testability, formalized test objectives, and fault-based testing techniques.