# Black-box Error Diagnosis in Deep Neural Networks for Computer Vision: a Survey of Tools

arXiv: https://arxiv.org/abs/2201.06444 · HTML: https://mlsanity.com/paper/2201.06444/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 95 references that are in the map.
- First posted on arXiv 17 Jan 2022.

## Builds on (most cited first, 30 of 40)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019)
- [Deep Learning for Anomaly Detection: A Survey](https://mlsanity.com/paper/1901.03407/) (2019)
- [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017)
- [The What-If Tool: Interactive Probing of Machine Learning Models](https://mlsanity.com/paper/1907.04135/) (2019)
- [Explainable Machine Learning for Scientific Insights and Discoveries](https://mlsanity.com/paper/1905.08883/) (2019)
- [Simultaneous Detection and Segmentation](https://mlsanity.com/paper/1407.1808/) (2014)
- [Deep Learning for Anomaly Detection: A Review](https://mlsanity.com/paper/2007.02500/) (2020)
- [Visual Interpretability for Deep Learning: a Survey](https://mlsanity.com/paper/1802.00614/) (2018)
- [RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms](https://mlsanity.com/paper/2011.01731/) (2020)
- [The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models](https://mlsanity.com/paper/2008.05122/) (2020)
- [TIDE: A General Toolbox for Identifying Object Detection Errors](https://mlsanity.com/paper/2008.08115/) (2020)
- [Darts: User-Friendly Modern Machine Learning for Time Series](https://mlsanity.com/paper/2110.03224/) (2021)
- [Black-box Explanation of Object Detectors via Saliency Maps](https://mlsanity.com/paper/2006.03204/) (2020)
- [Diagnosing Error in Temporal Action Detectors](https://mlsanity.com/paper/1807.10706/) (2018)
- [Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models](https://mlsanity.com/paper/1808.00196/) (2018)
- [Benchmarking and Error Diagnosis in Multi-Instance Pose Estimation](https://mlsanity.com/paper/1707.05388/) (2017)
- [Visual Analytics for Explainable Deep Learning](https://mlsanity.com/paper/1804.02527/) (2018)
- [How Far are We from Solving Pedestrian Detection?](https://mlsanity.com/paper/1602.01237/) (2016)
- [What Actions are Needed for Understanding Human Actions in Videos?](https://mlsanity.com/paper/1708.02696/) (2017)
- [Boxer: Interactive Comparison of Classifier Results](https://mlsanity.com/paper/2004.07964/) (2020)
- [Elliot: a Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation](https://mlsanity.com/paper/2103.02590/) (2021)
- [A Workflow for Visual Diagnostics of Binary Classifiers using Instance-Level Explanations](https://mlsanity.com/paper/1705.01968/) (2017)

## Built on by (most cited first, 1 of 1)

- [ChannelExplorer: Exploring Class Separability Through Activation Channel Visualization](https://mlsanity.com/paper/2505.04647/) (2025)

## Abstract (arXiv)

The application of Deep Neural Networks (DNNs) to a broad variety of tasks demands methods for coping with the complex and opaque nature of these architectures. When a gold standard is available, performance assessment treats the DNN as a black box and computes standard metrics based on the comparison of the predictions with the ground truth. A deeper understanding of performances requires going beyond such evaluation metrics to diagnose the model behavior and the prediction errors. This goal can be pursued in two complementary ways. On one side, model interpretation techniques "open the box" and assess the relationship between the input, the inner layers and the output, so as to identify the architecture modules most likely to cause the performance loss. On the other hand, black-box error diagnosis techniques study the correlation between the model response and some properties of the input not used for training, so as to identify the features of the inputs that make the model fail. Both approaches give hints on how to improve the architecture and/or the training process. This paper focuses on the application of DNNs to Computer Vision (CV) tasks and presents a survey of the tools that support the black-box performance diagnosis paradigm. It illustrates the features and gaps of the current proposals, discusses the relevant research directions and provides a brief overview of the diagnosis tools in sectors other than CV.