# Challenges in Testing Large Language Model Based Software: A Faceted Taxonomy arXiv: https://arxiv.org/abs/2503.00481 ยท HTML: https://mlsanity.com/paper/2503.00481/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 20 of its 60 references that are in the map. - First posted on arXiv 1 Mar 2025. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 20 of 20) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023) - [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023) - [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024) - [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023) - [Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning](https://mlsanity.com/paper/2310.03094/) (2023) - [Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction](https://mlsanity.com/paper/2209.11515/) (2022) - [Non-Determinism of "Deterministic" LLM Settings](https://mlsanity.com/paper/2408.04667/) (2024) - [RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models](https://mlsanity.com/paper/2409.19886/) (2024) - [A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization](https://mlsanity.com/paper/2308.05487/) (2023) - [On Testing Machine Learning Programs](https://mlsanity.com/paper/1812.02257/) (2018) - [Explainable Automated Debugging via Large Language Model-driven Scientific Debugging](https://mlsanity.com/paper/2304.02195/) (2023) - [An Integrative Survey on Mental Health Conversational Agents to Bridge Computer Science and Medical Perspectives](https://mlsanity.com/paper/2310.17017/) (2023) - [Evaluating Diverse Large Language Models for Automatic and General Bug Reproduction](https://mlsanity.com/paper/2311.04532/) (2023) - [Test Oracle Automation in the era of LLMs](https://mlsanity.com/paper/2405.12766/) (2024) - [Generating and Detecting True Ambiguity: A Forgotten Danger in DNN Supervision Testing](https://mlsanity.com/paper/2207.10495/) (2022) - [Automating a Complete Software Test Process Using LLMs: An Automotive Case Study](https://mlsanity.com/paper/2502.04008/) (2025) - [GoNoGo: An Efficient LLM-based Multi-Agent System for Streamlining Automotive Software Release Decision-Making](https://mlsanity.com/paper/2408.09785/) (2024) - [A Software Engineering Perspective on Testing Large Language Models: Research, Practice, Tools and Benchmarks](https://mlsanity.com/paper/2406.08216/) (2024) - [Assessing the Robustness of LLM-based NLP Software via Automated Testing](https://mlsanity.com/paper/2412.21016/) (2024) ## Built on by (most cited first, 8 of 8) - [An Empirical Study of Testing Practices in Open Source AI Agent Frameworks and Agentic Applications](https://mlsanity.com/paper/2509.19185/) (2025) - [Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications](https://mlsanity.com/paper/2603.15676/) (2026) - [The Semi-Executable Stack: Agentic Software Engineering and the Expanding Scope of SE](https://mlsanity.com/paper/2604.15468/) (2026) - [Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications](https://mlsanity.com/paper/2608.08413/) (2026) - [The Stability Trap: Evaluating the Reliability of LLM-Based Instruction Adherence Auditing](https://mlsanity.com/paper/2601.11783/) (2026) - [AVISE: Framework for Evaluating the Security of AI Systems](https://mlsanity.com/paper/2604.20833/) (2026) - [SenWorld: A Digital-Twin Simulation for Generating Context-Rich Evaluation Data](https://mlsanity.com/paper/2607.19949/) (2026) - [Testing and Evaluation of Agentic AI Systems In Military Command and Control](https://mlsanity.com/paper/2608.20597/) (2026) ## Abstract (arXiv) Large Language Models (LLMs) and Multi-Agent LLMs (MALLMs) introduce non-determinism unlike traditional or machine learning software, requiring new approaches to verifying correctness beyond simple output comparisons or statistical accuracy over test datasets. This paper presents a taxonomy for LLM test case design, informed by research literature and our experience. Each facet is exemplified, and we conduct an LLM-assisted analysis of six open-source testing frameworks, perform a sensitivity study of an agent-based system across different model configurations, and provide working examples contrasting atomic and aggregated test cases. We identify key variation points that impact test correctness and highlight open challenges that the research, industry, and open-source communities must address as LLMs become integral to software systems. Our taxonomy defines four facets of LLM test case design, addressing ambiguity in both inputs and outputs while establishing best practices. It distinguishes variability in goals, the system under test, and inputs, and introduces two key oracle types: atomic and aggregated. Our findings reveal that current tools treat test executions as isolated events, lack explicit aggregation mechanisms, and inadequately capture variability across model versions, configurations, and repeated runs. This highlights the need for viewing correctness as a distribution of outcomes rather than a binary property, requiring closer collaboration between academia and practitioners to establish mature, variability-aware testing methodologies.