# Test Set Diameter: Quantifying the Diversity of Sets of Test Cases

arXiv: https://arxiv.org/abs/1506.03482 · HTML: https://mlsanity.com/paper/1506.03482/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2015.

## Built on by (most cited first, 10 of 10)

- [Black-Box Testing of Deep Neural Networks Through Test Case Diversity](https://mlsanity.com/paper/2112.12591/) (2021)
- [Towards Autonomous Testing Agents via Conversational Large Language Models](https://mlsanity.com/paper/2306.05152/) (2023)
- [DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks](https://mlsanity.com/paper/2303.04878/) (2023)
- [Learning How to Search: Generating Effective Test Cases Through Adaptive Fitness Function Selection](https://mlsanity.com/paper/2102.04822/) (2021)
- [Test2Vec: An Execution Trace Embedding for Test Case Prioritization](https://mlsanity.com/paper/2206.15428/) (2022)
- [Reasoning-Based Software Testing](https://mlsanity.com/paper/2303.01302/) (2023)
- [Can Search-Based Testing with Pareto Optimization Effectively Cover Failure-Revealing Test Inputs?](https://mlsanity.com/paper/2410.11769/) (2024)
- [Adaptive Testing for LLM-Based Applications: A Diversity-based Approach](https://mlsanity.com/paper/2501.13480/) (2025)
- [Automated Support for Unit Test Generation: A Tutorial Book Chapter](https://mlsanity.com/paper/2110.13575/) (2021)
- [Search-based Testing of Vision Language Models for In-Car Scene Understanding](https://mlsanity.com/paper/2607.02300/) (2026)

## Abstract (arXiv)

A common and natural intuition among software testers is that test cases need to differ if a software system is to be tested properly and its quality ensured. Consequently, much research has gone into formulating distance measures for how test cases, their inputs and/or their outputs differ. However, common to these proposals is that they are data type specific and/or calculate the diversity only between pairs of test inputs, traces or outputs. We propose a new metric to measure the diversity of sets of tests: the test set diameter (TSDm). It extends our earlier, pairwise test diversity metrics based on recent advances in information theory regarding the calculation of the normalized compression distance (NCD) for multisets. An advantage is that TSDm can be applied regardless of data type and on any test-related information, not only the test inputs. A downside is the increased computational time compared to competing approaches. Our experiments on four different systems show that the test set diameter can help select test sets with higher structural and fault coverage than random selection even when only applied to test inputs. This can enable early test design and selection, prior to even having a software system to test, and complement other types of test automation and analysis. We argue that this quantification of test set diversity creates a number of opportunities to better understand software quality and provides practical ways to increase it.