# Exploring ML testing in practice -- Lessons learned from an interactive rapid review with Axis Communications

arXiv: https://arxiv.org/abs/2203.16225 · HTML: https://mlsanity.com/paper/2203.16225/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 52 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Builds on (most cited first, 22 of 22)

- [Identifying Mislabeled Data using the Area Under the Margin Ranking](https://mlsanity.com/paper/2001.10528/) (2020)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Guiding Deep Learning System Testing using Surprise Adequacy](https://mlsanity.com/paper/1808.08444/) (2018)
- [DeepMutation: Mutation Testing of Deep Learning Systems](https://mlsanity.com/paper/1805.05206/) (2018)
- [Requirements Engineering for Machine Learning: Perspectives from Data Scientists](https://mlsanity.com/paper/1908.04674/) (2019)
- [Towards Corner Case Detection for Autonomous Driving](https://mlsanity.com/paper/1902.09184/) (2019)
- [Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry](https://mlsanity.com/paper/1812.05389/) (2018)
- [Engineering AI Systems: A Research Agenda](https://mlsanity.com/paper/2001.07522/) (2020)
- [Reducing DNN Labelling Cost using Surprise Adequacy: An Industrial Case Study for Autonomous Driving](https://mlsanity.com/paper/2006.00894/) (2020)
- [Towards Structured Evaluation of Deep Neural Network Supervisors](https://mlsanity.com/paper/1903.01263/) (2019)
- [Towards Dependability Metrics for Neural Networks](https://mlsanity.com/paper/1806.02338/) (2018)
- [Input Prioritization for Testing Neural Networks](https://mlsanity.com/paper/1901.03768/) (2019)
- [Efficient and Effective Generation of Test Cases for Pedestrian Detection -- Search-based Software Testing of Baidu Apollo in SVL](https://mlsanity.com/paper/2109.07960/) (2021)
- [Exploring the Assessment List for Trustworthy AI in the Context of Advanced Driver-Assistance Systems](https://mlsanity.com/paper/2103.09051/) (2021)
- [Certification of embedded systems based on Machine Learning: A survey](https://mlsanity.com/paper/2106.07221/) (2021)
- [Rapid Reviews in Software Engineering](https://mlsanity.com/paper/2003.10006/) (2020)
- [The AIQ Meta-Testbed: Pragmatically Bridging Academic AI Testing and Industrial Q Needs](https://mlsanity.com/paper/2009.05260/) (2020)
- [Agility in Software 2.0 -- Notebook Interfaces and MLOps with Buttresses and Rebars](https://mlsanity.com/paper/2111.14142/) (2021)
- [Test Automation with Grad-CAM Heatmaps -- A Future Pipe Segment in MLOps for Vision AI?](https://mlsanity.com/paper/2103.01837/) (2021)
- [A Systematic Mapping Study on Testing of Machine Learning Programs](https://mlsanity.com/paper/1907.09427/) (2019)
- [On Testing Data-Intensive Software Systems](https://mlsanity.com/paper/1903.09413/) (2019)

## Built on by (most cited first, 3 of 3)

- [Ergo, SMIRK is Safe: A Safety Case for a Machine Learning Component in a Pedestrian Automatic Emergency Brake System](https://mlsanity.com/paper/2204.07874/) (2022)
- [A Conceptual Framework for Ethical Evaluation of Machine Learning Systems](https://mlsanity.com/paper/2408.10239/) (2024)
- [Beyond Accuracy: An Empirical Study on Unit Testing in Open-source Deep Learning Projects](https://mlsanity.com/paper/2402.16546/) (2024)

## Abstract (arXiv)

There is a growing interest in industry and academia in machine learning (ML) testing. We believe that industry and academia need to learn together to produce rigorous and relevant knowledge. In this study, we initiate a collaboration between stakeholders from one case company, one research institute, and one university. To establish a common view of the problem domain, we applied an interactive rapid review of the state of the art. Four researchers from Lund University and RISE Research Institutes and four practitioners from Axis Communications reviewed a set of 180 primary studies on ML testing. We developed a taxonomy for the communication around ML testing challenges and results and identified a list of 12 review questions relevant for Axis Communications. The three most important questions (data testing, metrics for assessment, and test generation) were mapped to the literature, and an in-depth analysis of the 35 primary studies matching the most important question (data testing) was made. A final set of the five best matches were analysed and we reflect on the criteria for applicability and relevance for the industry. The taxonomies are helpful for communication but not final. Furthermore, there was no perfect match to the case company's investigated review question (data testing). However, we extracted relevant approaches from the five studies on a conceptual level to support later context-specific improvements. We found the interactive rapid review approach useful for triggering and aligning communication between the different stakeholders.