# Quantitative Projection Coverage for Testing ML-enabled Autonomous Systems

arXiv: https://arxiv.org/abs/1805.04333 · HTML: https://mlsanity.com/paper/1805.04333/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2018.

## Built on by (most cited first, 11 of 11)

- [Machine Learning with Requirements: a Manifesto](https://mlsanity.com/paper/2304.03674/) (2023)
- [Toward Certification of Machine-Learning Systems for Low Criticality Airborne Applications](https://mlsanity.com/paper/2209.13975/) (2022)
- [Unaligned but Safe -- Formally Compensating Performance Limitations for Imprecise 2D Object Detection](https://mlsanity.com/paper/2202.05123/) (2022)
- [Prioritizing Corners in OoD Detectors via Symbolic String Manipulation](https://mlsanity.com/paper/2205.07736/) (2022)
- [Probabilistic Risk Assessment of an Obstacle Detection System for GoA 4 Freight Trains](https://mlsanity.com/paper/2306.14814/) (2023)
- [A Survey of Neural Network Robustness Assessment in Image Recognition](https://mlsanity.com/paper/2404.08285/) (2024)
- [Backdoor Mitigation in Deep Neural Networks via Strategic Retraining](https://mlsanity.com/paper/2212.07278/) (2022)
- [CLAP: Contrastive Latent-space Prompt Optimization for End-to-end Autonomous Driving](https://mlsanity.com/paper/2605.17284/) (2026)
- [Testing Autonomous Systems with Believed Equivalence Refinement](https://mlsanity.com/paper/2103.04578/) (2021)
- [Safe AI -- How is this Possible?](https://mlsanity.com/paper/2201.10436/) (2022)
- [Effective Black Box Testing of Sentiment Analysis Classification Networks](https://mlsanity.com/paper/2407.20884/) (2024)

## Abstract (arXiv)

Systematically testing models learned from neural networks remains a crucial unsolved barrier to successfully justify safety for autonomous vehicles engineered using data-driven approach. We propose quantitative k-projection coverage as a metric to mediate combinatorial explosion while guiding the data sampling process. By assuming that domain experts propose largely independent environment conditions and by associating elements in each condition with weights, the product of these conditions forms scenarios, and one may interpret weights associated with each equivalence class as relative importance. Achieving full k-projection coverage requires that the data set, when being projected to the hyperplane formed by arbitrarily selected k-conditions, covers each class with number of data points no less than the associated weight. For the general case where scenario composition is constrained by rules, precisely computing k-projection coverage remains in NP. In terms of finding minimum test cases to achieve full coverage, we present theoretical complexity for important sub-cases and an encoding to 0-1 integer programming. We have implemented a research prototype that generates test cases for a visual object defection unit in automated driving, demonstrating the technological feasibility of our proposed coverage criterion.