# Adaptive Testing for LLM-Based Applications: A Diversity-based Approach

arXiv: https://arxiv.org/abs/2501.13480 · HTML: https://mlsanity.com/paper/2501.13480/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 32 references that are in the map.
- First posted on arXiv 23 Jan 2025.

## Builds on (most cited first, 10 of 10)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences](https://mlsanity.com/paper/2404.12272/) (2024)
- [ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing](https://mlsanity.com/paper/2309.09128/) (2023)
- [LARP: Language-Agent Role Play for Open-World Games](https://mlsanity.com/paper/2312.17653/) (2023)
- [SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines](https://mlsanity.com/paper/2401.03038/) (2024)
- [Test Set Diameter: Quantifying the Diversity of Sets of Test Cases](https://mlsanity.com/paper/1506.03482/) (2015)
- [A Software Engineering Perspective on Testing Large Language Models: Research, Practice, Tools and Benchmarks](https://mlsanity.com/paper/2406.08216/) (2024)
- [Demystifying Issues, Causes and Solutions in LLM Open-Source Projects](https://mlsanity.com/paper/2409.16559/) (2024)

## Built on by (most cited first, 1 of 1)

- [Characterizing the Robustness of Black-Box LLM Planners Under Perturbed Observations with Adaptive Stress Testing](https://mlsanity.com/paper/2505.05665/) (2025)

## Abstract (arXiv)

The recent surge of building software systems powered by Large Language Models (LLMs) has led to the development of various testing frameworks, primarily focused on treating prompt templates as the unit of testing. Despite the significant costs associated with test input execution and output assessment, the curation of optimized test suites is yet overlooked in these tools, which calls for tailored test selection or prioritization strategies. In this paper, we show that diversity-based testing techniques, such as Adaptive Random Testing (ART) with appropriate string distance metrics, can be effectively applied to the testing of prompt templates. Our proposed adaptive testing approach adjusts the conventional ART process to this context by selecting new test inputs based on scores derived from existing test suite and their labelling results. Our results, obtained using various implementations that explore several string-based distances, confirm that our approach enables the discovery of failures with reduced testing budgets and promotes the generation of more varied outputs.