# Evaluating Deep Neural Networks in Deployment (A Comparative and Replicability Study)

arXiv: https://arxiv.org/abs/2407.08730 · HTML: https://mlsanity.com/paper/2407.08730/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jul 2024.

## Built on by (most cited first, 1 of 1)

- [Prophecy: Inferring Formal Properties from Neuron Activations](https://mlsanity.com/paper/2509.21677/) (2025)

## Abstract (arXiv)

As deep neural networks (DNNs) are increasingly used in safety-critical applications, there is a growing concern for their reliability. Even highly trained, high-performant networks are not 100% accurate. However, it is very difficult to predict their behavior during deployment without ground truth. In this paper, we provide a comparative and replicability study on recent approaches that have been proposed to evaluate the reliability of DNNs in deployment. We find that it is hard to run and reproduce the results for these approaches on their replication packages and even more difficult to run them on artifacts other than their own. Further, it is difficult to compare the effectiveness of the approaches, due to the lack of clearly defined evaluation metrics. Our results indicate that more effort is needed in our research community to obtain sound techniques for evaluating the reliability of neural networks in safety-critical domains. To this end, we contribute an evaluation framework that incorporates the considered approaches and enables evaluation on common benchmarks, using common metrics.