mlsanityData to 7 Oct 2026

A Survey of Large-Scale Deep Learning Serving System Optimization: Challenges and Opportunities

arXiv 2111.14247 ↗First posted 28 Nov 2021cs.LG cs.DCcited by 5

Not on the map. Why?

Abstract (arXiv)

Deep Learning (DL) models have achieved superior performance in many application domains, including vision, language, medical, commercial ads, entertainment, etc. With the fast development, both DL applications and the underlying serving hardware have demonstrated strong scaling trends, i.e., Model Scaling and Compute Scaling, for example, the recent pre-trained model with hundreds of billions of parameters with ~TB level memory consumption, as well as the newest GPU accelerators providing hundreds of TFLOPS. With both scaling trends, new problems and challenges emerge in DL inference serving systems, which gradually trends towards Large-scale Deep learning Serving systems (LDS). This survey aims to summarize and categorize the emerging challenges and optimization opportunities for large-scale deep learning serving systems. By providing a novel taxonomy, summarizing the computing paradigms, and elaborating the recent technique advances, we hope that this survey could shed light on new optimization perspectives and motivate novel works in large-scale deep learning system optimization.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 31 of its 54 references that are in the map. First posted on arXiv 28 Nov 2021.

Builds on

31 of 54 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  6. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  9. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  10. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  11. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  12. 2017Deep Learning for Medical Image Analysiscited by 399
19 more
  1. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  2. 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 291
  3. 2020Ansor: Generating High-Performance Tensor Programs for Deep Learningcited by 199
  4. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
  5. 2020CoEdge: Cooperative DNN Inference with Adaptive Workload Partitioning over Heterogeneous Edge Devicescited by 65
  6. 2018TVM: An Automated End-to-End Optimizing Compiler for Deep Learningcited by 62
  7. 2019Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Searchcited by 39
  8. 2021RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inferencecited by 35
  9. 2017Deep Learning for Video Game Playingcited by 30
  10. 2019PREMA: A Predictive Multi-task Scheduling Algorithm For Preemptible Neural Processing Unitscited by 30
  11. 2020Understanding Capacity-Driven Scale-Out Neural Recommendation Inferencecited by 29
  12. 2018Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networkscited by 23
  13. 2019EmBench: Quantifying Performance Variations of Deep Neural Networks across Modern Commodity Devicescited by 21
  14. 2020IOS: Inter-Operator Scheduler for CNN Accelerationcited by 20
  15. 2020Cortex: A Compiler for Recursive Deep Learning Modelscited by 16
  16. 2019Summarizing CPU and GPU Design Trends with Product Datacited by 10
  17. 2017Use of Deep Learning in Modern Recommendation System: A Summary of Recent Workscited by 5
  18. 2018Synergy: A HW/SW Framework for High Throughput CNNs on Embedded Heterogeneous SoCcited by 4
  19. 2021Multi-model Machine Learning Inference Serving with GPU Spatial Partitioningcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 1; 2025: 1.

5 papers.

  1. 2023DyCL: Dynamic Neural Network Compilation Via Program Rewriting and Graph Optimizationcited by 9
  2. 2022Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Visioncited by 6
  3. 2025Your Compiler is Backdooring Your Model: Understanding and Exploiting Compilation Inconsistency Vulnerabilities in Deep Learning Compilerscited by 6
  4. 2024Achieving Peak Performance for Large Language Models: A Systematic Reviewcited by 4
  5. 2022Improving the Performance of DNN-based Software Services using Automated Layer Cachingcited by 0