# Supporting Safety Analysis of Image-processing DNNs through Clustering-based Approaches

arXiv: https://arxiv.org/abs/2301.13506 · HTML: https://mlsanity.com/paper/2301.13506/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 108 references that are in the map.
- First posted on arXiv 31 Jan 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [Explaining NonLinear Classification Decisions with Deep Taylor Decomposition](https://mlsanity.com/paper/1512.02479/) (2015)
- [Real Time Image Saliency for Black Box Classifiers](https://mlsanity.com/paper/1705.07857/) (2017)
- [The ApolloScape Open Dataset for Autonomous Driving and its Application](https://mlsanity.com/paper/1803.06184/) (2018)
- [A Tutorial on Principal Component Analysis](https://mlsanity.com/paper/1404.1100/) (2014)
- [ApolloCar3D: A Large 3D Car Instance Understanding Benchmark for Autonomous Driving](https://mlsanity.com/paper/1811.12222/) (2018)
- [Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testing](https://mlsanity.com/paper/2007.02787/) (2020)
- [Can Offline Testing of Deep Neural Networks Replace Their Online Testing?](https://mlsanity.com/paper/2101.11118/) (2021)
- [Scale-aware Automatic Augmentation for Object Detection](https://mlsanity.com/paper/2103.17220/) (2021)
- [SVIRO: Synthetic Vehicle Interior Rear Seat Occupancy Dataset and Benchmark](https://mlsanity.com/paper/2001.03483/) (2020)
- [Self-Driving Car Steering Angle Prediction Based on Image Recognition](https://mlsanity.com/paper/1912.05440/) (2019)
- [Automated Classification of Histopathology Images Using Transfer Learning](https://mlsanity.com/paper/1903.10035/) (2019)
- [Supporting DNN Safety Analysis and Retraining through Heatmap-based Unsupervised Learning](https://mlsanity.com/paper/2002.00863/) (2020)
- [Automatic Test Suite Generation for Key-Points Detection DNNs using Many-Objective Search (Experience Paper)](https://mlsanity.com/paper/2012.06511/) (2020)
- [Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction and Clustering](https://mlsanity.com/paper/2201.05077/) (2022)

## Built on by (most cited first, 2 of 2)

- [Bridging the Gap between Real-world and Synthetic Images for Testing Autonomous Driving Systems](https://mlsanity.com/paper/2408.13950/) (2024)
- [GAN-enhanced Simulation-driven DNN Testing in Absence of Ground Truth](https://mlsanity.com/paper/2503.15953/) (2025)

## Abstract (arXiv)

The adoption of deep neural networks (DNNs) in safety-critical contexts is often prevented by the lack of effective means to explain their results, especially when they are erroneous. In our previous work, we proposed a white-box approach (HUDD) and a black-box approach (SAFE) to automatically characterize DNN failures. They both identify clusters of similar images from a potentially large set of images leading to DNN failures. However, the analysis pipelines for HUDD and SAFE were instantiated in specific ways according to common practices, deferring the analysis of other pipelines to future work. In this paper, we report on an empirical evaluation of 99 different pipelines for root cause analysis of DNN failures. They combine transfer learning, autoencoders, heatmaps of neuron relevance, dimensionality reduction techniques, and different clustering algorithms. Our results show that the best pipeline combines transfer learning, DBSCAN, and UMAP. It leads to clusters almost exclusively capturing images of the same failure scenario, thus facilitating root cause analysis. Further, it generates distinct clusters for each root cause of failure, thus enabling engineers to detect all the unsafe scenarios. Interestingly, these results hold even for failure scenarios that are only observed in a small percentage of the failing images.