# Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol

arXiv: https://arxiv.org/abs/2508.20737 · HTML: https://mlsanity.com/paper/2508.20737/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 25 of its 77 references that are in the map.
- First posted on arXiv 28 Aug 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 25 of 25)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://mlsanity.com/paper/2310.04451/) (2023)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [Survey on Evaluation of LLM-based Agents](https://mlsanity.com/paper/2503.16416/) (2025)
- [Efficient Adversarial Training in LLMs with Continuous Attacks](https://mlsanity.com/paper/2405.15589/) (2024)
- [Assessing Adversarial Robustness of Large Language Models: An Empirical Study](https://mlsanity.com/paper/2405.02764/) (2024)
- [PDL: A Declarative Prompt Programming Language](https://mlsanity.com/paper/2410.19135/) (2024)
- [The challenge of uncertainty quantification of large language models in medicine](https://mlsanity.com/paper/2504.05278/) (2025)
- [DeepRoad: GAN-based Metamorphic Autonomous Driving System Testing](https://mlsanity.com/paper/1802.02295/) (2018)
- [Brain-inspired Artificial Intelligence: A Comprehensive Review](https://mlsanity.com/paper/2408.14811/) (2024)
- [MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systems](https://mlsanity.com/paper/2412.15557/) (2024)
- [Risks When Sharing LoRA Fine-Tuned Diffusion Model Weights](https://mlsanity.com/paper/2409.08482/) (2024)
- [Boosting Backdoor Attack with A Learnable Poisoning Sample Selection Strategy](https://mlsanity.com/paper/2307.07328/) (2023)
- [Comparing Template-based and Template-free Language Model Probing](https://mlsanity.com/paper/2402.00123/) (2024)
- [Open Source Software Development Tool Installation: Challenges and Strategies For Novice Developers](https://mlsanity.com/paper/2404.14637/) (2024)
- [The Polynomial Stein Discrepancy for Assessing Moment Convergence](https://mlsanity.com/paper/2412.05135/) (2024)
- [Predicting Future States with Spatial Point Processes in Single Molecule Resolution Spatial Transcriptomics](https://mlsanity.com/paper/2401.02564/) (2024)
- [RT-DEMT: A hybrid real-time acupoint detection model combining mamba and transformer](https://mlsanity.com/paper/2502.11179/) (2025)
- [Demystifying AI Agents: The Final Generation of Intelligence](https://mlsanity.com/paper/2505.09932/) (2025)

## Built on by (most cited first, 6 of 6)

- [An Empirical Study of Testing Practices in Open Source AI Agent Frameworks and Agentic Applications](https://mlsanity.com/paper/2509.19185/) (2025)
- [Agent Alpha: Tree Search Unifying Generation, Exploration and Evaluation for Computer-Use Agents](https://mlsanity.com/paper/2602.02995/) (2026)
- [How to Interpret Agent Behavior](https://mlsanity.com/paper/2605.13625/) (2026)
- [AAFLOW: Scalable Patterns for Agentic AI Workflows](https://mlsanity.com/paper/2605.02162/) (2026)
- [Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications](https://mlsanity.com/paper/2608.08413/) (2026)
- [LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications](https://mlsanity.com/paper/2603.27355/) (2026)

## Abstract (arXiv)

Applications of Large Language Models~(LLMs) have evolved from simple text generators into complex software systems that integrate retrieval augmentation, tool invocation, and multi-turn interactions. Their inherent non-determinism, dynamism, and context dependence pose fundamental challenges for quality assurance. This paper decomposes LLM applications into a three-layer architecture: textbftextitSystem Shell Layer, textbftextitPrompt Orchestration Layer, and textbftextitLLM Inference Core. We then assess the applicability of traditional software testing methods in each layer: directly applicable at the shell layer, requiring semantic reinterpretation at the orchestration layer, and necessitating paradigm shifts at the inference core. A comparative analysis of Testing AI methods from the software engineering community and safety analysis techniques from the AI community reveals structural disconnects in testing unit abstraction, evaluation metrics, and lifecycle management. We identify four fundamental differences that underlie 6 core challenges. To address these, we propose four types of collaborative strategies (Retain, Translate, Integrate, and Runtime) and explore a closed-loop, trustworthy quality assurance framework that combines pre-deployment validation with runtime monitoring. Based on these strategies, we offer practical guidance and a protocol proposal to support the standardization and tooling of LLM application testing. We propose a protocol textbftextitAgent Interaction Communication Language (AICL) that is used to communicate between AI agents. AICL has the test-oriented features and is easily integrated in the current agent framework.