# Smoke Testing for Machine Learning: Simple Tests to Discover Severe Defects

arXiv: https://arxiv.org/abs/2009.01521 · HTML: https://mlsanity.com/paper/2009.01521/

## Key facts

- Cited by 5 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 3 Sep 2020.

## Built on by (most cited first, 5 of 5)

- [Deep Learning Library Testing: Definition, Methods and Challenges](https://mlsanity.com/paper/2404.17871/) (2024)
- [Differential testing for machine learning: an analysis for classification algorithms beyond deep learning](https://mlsanity.com/paper/2207.11976/) (2022)
- [Test & Evaluation Best Practices for Machine Learning-Enabled Systems](https://mlsanity.com/paper/2310.06800/) (2023)
- [A Review on Oracle Issues in Machine Learning](https://mlsanity.com/paper/2105.01407/) (2021)
- [Broken on Arrival: Silently Defective LLM Artifacts in Public Model Registries and How to Catch Them](https://mlsanity.com/paper/2609.05881/) (2026)

## Abstract (arXiv)

Machine learning is nowadays a standard technique for data analysis within software applications. Software engineers need quality assurance techniques that are suitable for these new kinds of systems. Within this article, we discuss the question whether standard software testing techniques that have been part of textbooks since decades are also useful for the testing of machine learning software. Concretely, we try to determine generic and simple smoke tests that can be used to assert that basic functions can be executed without crashing. We found that we can derive such tests using techniques similar to equivalence classes and boundary value analysis. Moreover, we found that these concepts can also be applied to hyperparameters, to further improve the quality of the smoke tests. Even though our approach is almost trivial, we were able to find bugs in all three machine learning libraries that we tested and severe bugs in two of the three libraries. This demonstrates that common software testing techniques are still valid in the age of machine learning and that considerations how they can be adapted to this new context can help to find and prevent severe bugs, even in mature machine learning libraries.