# Iterative Assessment and Improvement of DNN Operational Accuracy

arXiv: https://arxiv.org/abs/2303.01295 · HTML: https://mlsanity.com/paper/2303.01295/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 39 references that are in the map.
- First posted on arXiv 2 Mar 2023.

## Builds on (most cited first, 13 of 13)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Addressing Failure Prediction by Learning Model Confidence](https://mlsanity.com/paper/1910.04851/) (2019)
- [A Unified View of Label Shift Estimation](https://mlsanity.com/paper/2003.07554/) (2020)
- [DeepMutation: Mutation Testing of Deep Learning Systems](https://mlsanity.com/paper/1805.05206/) (2018)
- [Misbehaviour Prediction for Autonomous Driving Systems](https://mlsanity.com/paper/1910.04443/) (2019)
- [Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges](https://mlsanity.com/paper/1905.04223/) (2019)
- [Boosting Operational DNN Testing Efficiency through Conditioning](https://mlsanity.com/paper/1906.02533/) (2019)
- [Self-Checking Deep Neural Networks in Deployment](https://mlsanity.com/paper/2103.02371/) (2021)
- [Human vs. supervised machine learning: Who learns patterns faster?](https://mlsanity.com/paper/2012.03661/) (2020)
- [Operation is the hardest teacher: estimating DNN accuracy looking for mispredictions](https://mlsanity.com/paper/2102.04287/) (2021)

## Built on by (most cited first, 3 of 3)

- [DeepSample: DNN sampling-based testing for operational accuracy assessment](https://mlsanity.com/paper/2403.19271/) (2024)
- [xMem: A CPU-Based Approach for Accurate Estimation of GPU Memory in Deep Learning Training Workloads](https://mlsanity.com/paper/2510.21048/) (2025)
- [Accurate GPU Memory Prediction for Deep Learning Jobs through Dynamic Analysis](https://mlsanity.com/paper/2504.03887/) (2025)

## Abstract (arXiv)

Deep Neural Networks (DNN) are nowadays largely adopted in many application domains thanks to their human-like, or even superhuman, performance in specific tasks. However, due to unpredictable/unconsidered operating conditions, unexpected failures show up on field, making the performance of a DNN in operation very different from the one estimated prior to release. In the life cycle of DNN systems, the assessment of accuracy is typically addressed in two ways: offline, via sampling of operational inputs, or online, via pseudo-oracles. The former is considered more expensive due to the need for manual labeling of the sampled inputs. The latter is automatic but less accurate. We believe that emerging iterative industrial-strength life cycle models for Machine Learning systems, like MLOps, offer the possibility to leverage inputs observed in operation not only to provide faithful estimates of a DNN accuracy, but also to improve it through remodeling/retraining actions. We propose DAIC (DNN Assessment and Improvement Cycle), an approach which combines ''low-cost'' online pseudo-oracles and ''high-cost'' offline sampling techniques to estimate and improve the operational accuracy of a DNN in the iterations of its life cycle. Preliminary results show the benefits of combining the two approaches and integrating them in the DNN life cycle.