# A Comparative Measurement Study of Deep Learning as a Service Framework

arXiv: https://arxiv.org/abs/1810.12210 · HTML: https://mlsanity.com/paper/1810.12210/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2018.

## Built on by (most cited first, 10 of 10)

- [CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaboration](https://mlsanity.com/paper/2411.02829/) (2024)
- [Rethinking Learning Rate Tuning in the Era of Large Language Models](https://mlsanity.com/paper/2309.08859/) (2023)
- [DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models](https://mlsanity.com/paper/2409.06669/) (2024)
- [Selecting and Composing Learning Rate Policies for Deep Neural Networks](https://mlsanity.com/paper/2210.12936/) (2022)
- [Exploring Effects of Computational Parameter Changes to Image Recognition Systems](https://mlsanity.com/paper/2211.00471/) (2022)
- [DeltaNN: Assessing the Impact of Computational Environment Parameters on the Performance of Image Recognition Models](https://mlsanity.com/paper/2306.06208/) (2023)
- [On the Efficiency of Privacy Attacks in Federated Learning](https://mlsanity.com/paper/2404.09430/) (2024)
- [Data science and Machine learning in the Clouds: A Perspective for the Future](https://mlsanity.com/paper/2109.01661/) (2021)
- [An Overview of the Data-Loader Landscape: Comparative Performance Analysis](https://mlsanity.com/paper/2209.13705/) (2022)
- [Split-n-Chain: Privacy-Preserving Multi-Node Split Learning with Blockchain-Based Auditability](https://mlsanity.com/paper/2503.07570/) (2025)

## Abstract (arXiv)

Big data powered Deep Learning (DL) and its applications have blossomed in recent years, fueled by three technological trends: a large amount of digitized data openly accessible, a growing number of DL software frameworks in open source and commercial markets, and a selection of affordable parallel computing hardware devices. However, no single DL framework, to date, dominates in terms of performance and accuracy even for baseline classification tasks on standard datasets, making the selection of a DL framework an overwhelming task. This paper takes a holistic approach to conduct empirical comparison and analysis of four representative DL frameworks with three unique contributions. First, given a selection of CPU-GPU configurations, we show that for a specific DL framework, different configurations of its hyper-parameters may have a significant impact on both performance and accuracy of DL applications. Second, to the best of our knowledge, this study is the first to identify the opportunities for improving the training time performance and the accuracy of DL frameworks by configuring parallel computing libraries and tuning individual and multiple hyper-parameters. Third, we also conduct a comparative measurement study on the resource consumption patterns of four DL frameworks and their performance and accuracy implications, including CPU and memory usage, and their correlations to varying settings of hyper-parameters under different configuration combinations of hardware, parallel computing libraries. We argue that this measurement study provides in-depth empirical comparison and analysis of four representative DL frameworks, and offers practical guidance for service providers to deploying and delivering DL as a Service (DLaaS) and for application developers and DLaaS consumers to select the right DL frameworks for the right DL workloads.