# DeltaNN: Assessing the Impact of Computational Environment Parameters on the Performance of Image Recognition Models

arXiv: https://arxiv.org/abs/2306.06208 · HTML: https://mlsanity.com/paper/2306.06208/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 57 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [CoAtNet: Marrying Convolution and Attention for All Data Sizes](https://mlsanity.com/paper/2106.04803/) (2021)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [cuDNN: Efficient Primitives for Deep Learning](https://mlsanity.com/paper/1410.0759/) (2014)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Ansor: Generating High-Performance Tensor Programs for Deep Learning](https://mlsanity.com/paper/2006.06762/) (2020)
- [Taxonomy of Real Faults in Deep Learning Systems](https://mlsanity.com/paper/1910.11015/) (2019)
- [Benchmarking State-of-the-Art Deep Learning Software Tools](https://mlsanity.com/paper/1608.07249/) (2016)
- [On the Role of Text Preprocessing in Neural Network Architectures: An Evaluation Study on Text Categorization and Sentiment Analysis](https://mlsanity.com/paper/1707.01780/) (2017)
- [Characterising Across-Stack Optimisations for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1809.07196/) (2018)
- [A Comparative Measurement Study of Deep Learning as a Service Framework](https://mlsanity.com/paper/1810.12210/) (2018)
- [An Empirical Study of Challenges in Converting Deep Learning Models](https://mlsanity.com/paper/2206.14322/) (2022)
- [VERIFAI: A Toolkit for the Design and Analysis of Artificial Intelligence-Based Systems](https://mlsanity.com/paper/1902.04245/) (2019)
- [Transfer-Tuning: Reusing Auto-Schedules for Efficient Tensor Program Code Generation](https://mlsanity.com/paper/2201.05587/) (2022)
- [M3: Semantic API Migrations](https://mlsanity.com/paper/2008.12118/) (2020)
- [Exploring Effects of Computational Parameter Changes to Image Recognition Systems](https://mlsanity.com/paper/2211.00471/) (2022)
- [Improving Robustness Against Adversarial Attacks with Deeply Quantized Neural Networks](https://mlsanity.com/paper/2304.12829/) (2023)

## Built on by (most cited first, 3 of 3)

- [Deep Learning Library Testing: Definition, Methods and Challenges](https://mlsanity.com/paper/2404.17871/) (2024)
- [FetaFix: Automatic Fault Localization and Repair of Deep Learning Model Conversions](https://mlsanity.com/paper/2312.15101/) (2023)
- [On the Mistaken Assumption of Interchangeable Deep Reinforcement Learning Implementations](https://mlsanity.com/paper/2503.22575/) (2025)

## Abstract (arXiv)

Image recognition tasks typically use deep learning and require enormous processing power, thus relying on hardware accelerators like GPUs and TPUs for fast, timely processing. Failure in real-time image recognition tasks can occur due to sub-optimal mapping on hardware accelerators during model deployment, which may lead to timing uncertainty and erroneous behavior. Mapping on hardware accelerators is done using multiple software components like deep learning frameworks, compilers, and device libraries, that we refer to as the computational environment. Owing to the increased use of image recognition tasks in safety-critical applications like autonomous driving and medical imaging, it is imperative to assess their robustness to changes in the computational environment, as the impact of parameters like deep learning frameworks, compiler optimizations, and hardware devices on model performance and correctness is not yet well understood. In this paper we present a differential testing framework, DeltaNN, that allows us to assess the impact of different computational environment parameters on the performance of image recognition models during deployment, post training. DeltaNN generates different implementations of a given image recognition model for variations in environment parameters, namely, deep learning frameworks, compiler optimizations and hardware devices and analyzes differences in model performance as a result. Using DeltaNN, we conduct an empirical study of robustness analysis of three popular image recognition models using the ImageNet dataset. We report the impact in terms of misclassifications and inference time differences across different settings. In total, we observed up to 100% output label differences across deep learning frameworks, and up to 81% unexpected performance degradation in terms of inference time, when applying compiler optimizations.