# Reducing DNN Labelling Cost using Surprise Adequacy: An Industrial Case Study for Autonomous Driving

arXiv: https://arxiv.org/abs/2006.00894 · HTML: https://mlsanity.com/paper/2006.00894/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2020.

## Built on by (most cited first, 22 of 22)

- [Synthetic Datasets for Autonomous Driving: A Survey](https://mlsanity.com/paper/2304.12205/) (2023)
- [Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning (Replicability Study)](https://mlsanity.com/paper/2205.00664/) (2022)
- [When and Why Test Generators for Deep Learning Produce Invalid Inputs: an Empirical Study](https://mlsanity.com/paper/2212.11368/) (2022)
- [MDPFuzz: Testing Models Solving Markov Decision Processes](https://mlsanity.com/paper/2112.02807/) (2021)
- [Revisiting Neuron Coverage for DNN Testing: A Layer-Wise and Distribution-Aware Criterion](https://mlsanity.com/paper/2112.01955/) (2021)
- [A Review and Refinement of Surprise Adequacy](https://mlsanity.com/paper/2103.05939/) (2021)
- [Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction and Clustering](https://mlsanity.com/paper/2201.05077/) (2022)
- [Generating and Detecting True Ambiguity: A Forgotten Danger in DNN Supervision Testing](https://mlsanity.com/paper/2207.10495/) (2022)
- [Understanding the Effectiveness of Coverage Criteria for Large Language Models: A Special Angle from Jailbreak Attacks](https://mlsanity.com/paper/2408.15207/) (2024)
- [A Unified Approach Towards Active Learning and Out-of-Distribution Detection](https://mlsanity.com/paper/2405.11337/) (2024)
- [Corner case data description and detection](https://mlsanity.com/paper/2101.02494/) (2021)
- [Exploring ML testing in practice -- Lessons learned from an interactive rapid review with Axis Communications](https://mlsanity.com/paper/2203.16225/) (2022)
- [Uncertainty Quantification for Deep Neural Networks: An Empirical Comparison and Usage Guidelines](https://mlsanity.com/paper/2212.07118/) (2022)
- [Guiding the retraining of convolutional neural networks against adversarial inputs](https://mlsanity.com/paper/2207.03689/) (2022)
- [DANDI: Diffusion as Normative Distribution for Deep Neural Network Input](https://mlsanity.com/paper/2502.02910/) (2025)
- [Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs](https://mlsanity.com/paper/2509.17314/) (2025)
- [Anomaly Detection in Driving by Cluster Analysis Twice](https://mlsanity.com/paper/2212.07691/) (2022)
- [Adopting Two Supervisors for Efficient Use of Large-Scale Remote Deep Neural Networks](https://mlsanity.com/paper/2304.02654/) (2023)
- [GAN-enhanced Simulation-driven DNN Testing in Absence of Ground Truth](https://mlsanity.com/paper/2503.15953/) (2025)
- [TopoMap: A Feature-based Semantic Discriminator of the Topographical Regions in the Test Input Space](https://mlsanity.com/paper/2509.03242/) (2025)
- [RACC: Representation-Aware Coverage Criteria for LLM Safety Testing](https://mlsanity.com/paper/2602.02280/) (2026)
- [Test Case Prioritization for DNNs via Neural Collapse Instability](https://mlsanity.com/paper/2607.20046/) (2026)

## Abstract (arXiv)

Deep Neural Networks (DNNs) are rapidly being adopted by the automotive industry, due to their impressive performance in tasks that are essential for autonomous driving. Object segmentation is one such task: its aim is to precisely locate boundaries of objects and classify the identified objects, helping autonomous cars to recognise the road environment and the traffic situation. Not only is this task safety critical, but developing a DNN based object segmentation module presents a set of challenges that are significantly different from traditional development of safety critical software. The development process in use consists of multiple iterations of data collection, labelling, training, and evaluation. Among these stages, training and evaluation are computation intensive while data collection and labelling are manual labour intensive. This paper shows how development of DNN based object segmentation can be improved by exploiting the correlation between Surprise Adequacy (SA) and model performance. The correlation allows us to predict model performance for inputs without manually labelling them. This, in turn, enables understanding of model performance, more guided data collection, and informed decisions about further training. In our industrial case study the technique allows cost savings of up to 50% with negligible evaluation inaccuracy. Furthermore, engineers can trade off cost savings versus the tolerable level of inaccuracy depending on different development phases and scenarios.