# Systematic Training and Testing for Machine Learning Using Combinatorial Interaction Testing

arXiv: https://arxiv.org/abs/2201.12428 · HTML: https://mlsanity.com/paper/2201.12428/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 29 references that are in the map.
- First posted on arXiv 28 Jan 2022.

## Builds on (most cited first, 7 of 7)

- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems](https://mlsanity.com/paper/1803.07519/) (2018)
- [Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC](https://mlsanity.com/paper/1507.04544/) (2015)
- [On Testing Machine Learning Programs](https://mlsanity.com/paper/1812.02257/) (2018)
- [Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components](https://mlsanity.com/paper/1804.06760/) (2018)
- [FreaAI: Automated extraction of data slices to test machine learning models](https://mlsanity.com/paper/2108.05620/) (2021)

## Built on by (most cited first, 6 of 6)

- [Test & Evaluation Best Practices for Machine Learning-Enabled Systems](https://mlsanity.com/paper/2310.06800/) (2023)
- [Active Learning with Combinatorial Coverage](https://mlsanity.com/paper/2302.14567/) (2023)
- [On Extending the Automatic Test Markup Language (ATML) for Machine Learning](https://mlsanity.com/paper/2404.03769/) (2024)
- [Bridging the Data Gap in AI Reliability Research and Establishing DR-AIR, a Comprehensive Data Repository for AI Reliability](https://mlsanity.com/paper/2502.12386/) (2025)
- [Metric Learning Improves the Ability of Combinatorial Coverage Metrics to Anticipate Classification Error](https://mlsanity.com/paper/2302.14616/) (2023)
- [Underwater SONAR Image Classification and Analysis using LIME-based Explainable Artificial Intelligence](https://mlsanity.com/paper/2408.12837/) (2024)

## Abstract (arXiv)

This paper demonstrates the systematic use of combinatorial coverage for selecting and characterizing test and training sets for machine learning models. The presented work adapts combinatorial interaction testing, which has been successfully leveraged in identifying faults in software testing, to characterize data used in machine learning. The MNIST hand-written digits data is used to demonstrate that combinatorial coverage can be used to select test sets that stress machine learning model performance, to select training sets that lead to robust model performance, and to select data for fine-tuning models to new domains. Thus, the results posit combinatorial coverage as a holistic approach to training and testing for machine learning. In contrast to prior work which has focused on the use of coverage in regard to the internal of neural networks, this paper considers coverage over simple features derived from inputs and outputs. Thus, this paper addresses the case where the supplier of test and training sets for machine learning models does not have intellectual property rights to the models themselves. Finally, the paper addresses prior criticism of combinatorial coverage and provides a rebuttal which advocates the use of coverage metrics in machine learning applications.