# An Empirical Study towards Characterizing Deep Learning Development and Deployment across Different Frameworks and Platforms

arXiv: https://arxiv.org/abs/1909.06727 · HTML: https://mlsanity.com/paper/1909.06727/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2019.

## Built on by (most cited first, 30 of 35)

- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [Smart at what cost? Characterising Mobile Deep Neural Networks in the wild](https://mlsanity.com/paper/2109.13963/) (2021)
- [Towards Training Reproducible Deep Learning Models](https://mlsanity.com/paper/2202.02326/) (2022)
- [An Empirical Study on Deployment Faults of Deep Learning Based Mobile Applications](https://mlsanity.com/paper/2101.04930/) (2021)
- [Are LLMs Correctly Integrated into Software Systems?](https://mlsanity.com/paper/2407.05138/) (2024)
- [Understanding Performance Problems in Deep Learning Systems](https://mlsanity.com/paper/2112.01771/) (2021)
- [Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey](https://mlsanity.com/paper/2310.17903/) (2023)
- [Large Language Model Supply Chain: Open Problems From the Security Perspective](https://mlsanity.com/paper/2411.01604/) (2024)
- [An Empirical Study of Challenges in Converting Deep Learning Models](https://mlsanity.com/paper/2206.14322/) (2022)
- [Integration of Convolutional Neural Networks in Mobile Applications](https://mlsanity.com/paper/2103.07286/) (2021)
- [Which Design Decisions in AI-enabled Mobile Applications Contribute to Greener AI?](https://mlsanity.com/paper/2109.15284/) (2021)
- [Anatomizing Deep Learning Inference in Web Browsers](https://mlsanity.com/paper/2402.05981/) (2024)
- [Deep Learning Library Testing: Definition, Methods and Challenges](https://mlsanity.com/paper/2404.17871/) (2024)
- [Hidden Reliability Risks in Large Language Models: Systematic Identification of Precision-Induced Output Disagreements](https://mlsanity.com/paper/2604.19790/) (2026)
- [Test & Evaluation Best Practices for Machine Learning-Enabled Systems](https://mlsanity.com/paper/2310.06800/) (2023)
- [The challenge of reproducible ML: an empirical study on the impact of bugs](https://mlsanity.com/paper/2109.03991/) (2021)
- [Why Does the LLM Stop Computing: An Empirical Study of User-Reported Failures in Open-Source LLMs](https://mlsanity.com/paper/2601.13655/) (2026)
- [Bridging the Language Gap: An Empirical Study of Bindings for Open Source Machine Learning Libraries Across Software Package Ecosystems](https://mlsanity.com/paper/2201.07201/) (2022)
- [Characterizing and Understanding the Behavior of Quantized Models for Reliable Deployment](https://mlsanity.com/paper/2204.04220/) (2022)
- [Deploying Deep Reinforcement Learning Systems: A Taxonomy of Challenges](https://mlsanity.com/paper/2308.12438/) (2023)
- [ML-On-Rails: Safeguarding Machine Learning Models in Software Systems A Case Study](https://mlsanity.com/paper/2401.06513/) (2024)
- [Robustness Analysis of Deep Learning Frameworks on Mobile Platforms](https://mlsanity.com/paper/2109.09869/) (2021)
- [Finding Deviated Behaviors of the Compressed DNN Models for Image Classifications](https://mlsanity.com/paper/2112.02819/) (2021)
- [Supervised Robustness-preserving Data-free Neural Network Pruning](https://mlsanity.com/paper/2204.00783/) (2022)
- [Tightening Robustness Verification of MaxPool-based Neural Networks via Minimizing the Over-Approximation Zone](https://mlsanity.com/paper/2211.09810/) (2022)
- [Studying the Impact of TensorFlow and PyTorch Bindings on Machine Learning Software Quality](https://mlsanity.com/paper/2407.05466/) (2024)
- [Towards Migrating Neural Network Implementations](https://mlsanity.com/paper/2511.02610/) (2025)
- [Engineering an Intelligent Essay Scoring and Feedback System: An Experience Report](https://mlsanity.com/paper/2103.13590/) (2021)
- [On Causal Inference for Data-free Structured Pruning](https://mlsanity.com/paper/2112.10229/) (2021)
- [Quality at the Tail of Machine Learning Inference](https://mlsanity.com/paper/2212.13925/) (2022)

## Abstract (arXiv)

Deep Learning (DL) has recently achieved tremendous success. A variety of DL frameworks and platforms play a key role to catalyze such progress. However, the differences in architecture designs and implementations of existing frameworks and platforms bring new challenges for DL software development and deployment. Till now, there is no study on how various mainstream frameworks and platforms influence both DL software development and deployment in practice. To fill this gap, we take the first step towards understanding how the most widely-used DL frameworks and platforms support the DL software development and deployment. We conduct a systematic study on these frameworks and platforms by using two types of DNN architectures and three popular datasets. (1) For development process, we investigate the prediction accuracy under the same runtime training configuration or same model weights/biases. We also study the adversarial robustness of trained models by leveraging the existing adversarial attack techniques. The experimental results show that the computing differences across frameworks could result in an obvious prediction accuracy decline, which should draw the attention of DL developers. (2) For deployment process, we investigate the prediction accuracy and performance (refers to time cost and memory consumption) when the trained models are migrated/quantized from PC to real mobile devices and web browsers. The DL platform study unveils that the migration and quantization still suffer from compatibility and reliability issues. Meanwhile, we find several DL software bugs by using the results as a benchmark. We further validate the results through bug confirmation from stakeholders and industrial positive feedback to highlight the implications of our study. Through our study, we summarize practical guidelines, identify challenges and pinpoint new research directions.