mlsanityData to 14 Oct 2026

Towards Demystifying Serverless Machine Learning Training

arXiv 2105.07806 ↗First posted 17 May 2021cs.DC cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

The appeal of serverless (FaaS) has triggered a growing interest on how to use it in data-intensive applications such as ETL, query processing, or machine learning (ML). Several systems exist for training large-scale ML models on top of serverless infrastructures (e.g., AWS Lambda) but with inconclusive results in terms of their performance and relative advantage over "serverful" infrastructures (IaaS). In this paper we present a systematic, comparative study of distributed ML training over FaaS and IaaS. We present a design space covering design choices such as optimization algorithms and synchronization protocols, and implement a platform, LambdaML, that enables a fair comparison between FaaS and IaaS. We present experimental results using LambdaML, and further develop an analytic model to capture cost/performance tradeoffs that must be considered when opting for a serverless infrastructure. Our results indicate that ML training pays off in serverless only for models with efficient (i.e., reduced) communication and that quickly converge. In general, FaaS can be much faster but it is never significantly cheaper than IaaS.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 39 of its 104 references that are in the map. First posted on arXiv 17 May 2021.

Builds on

39 of 104 references.

References · 39

  1. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  2. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  3. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  4. 2018The Convergence of Sparsified Gradient Methodscited by 229
  5. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 221
  6. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  7. 2013On the Convergence of Decentralized Gradient Descentcited by 159
  8. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  9. 2018D2: Decentralized Training over Decentralized Datacited by 140
  10. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 136
  11. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 99
  12. 2018Communication Compression for Decentralized Trainingcited by 88
27 more
  1. 2018Byzantine Stochastic Gradient Descentcited by 81
  2. 2018Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGDcited by 77
  3. 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
  4. 2016CoCoA: A General Framework for Communication-Efficient Distributed Optimizationcited by 69
  5. 2015MLlib: Machine Learning in Apache Sparkcited by 55
  6. 2018COLA: Decentralized Linear Learningcited by 30
  7. 2018TicTac: Accelerating Distributed Deep Learning with Communication Schedulingcited by 27
  8. 2017Straggler Mitigation in Distributed Optimization Through Data Encodingcited by 26
  9. 2016Gossip Dual Averaging for Decentralized Optimization of Pairwise Functionscited by 21
  10. 2015Adding vs. Averaging in Distributed Primal-Dual Optimizationcited by 19
  11. 2015Asynchronous stochastic convex optimizationcited by 19
  12. 2017Serverless Computing: Current Trends and Open Problemscited by 17
  13. 2021Distributed Learning Systems with First-order Methodscited by 17
  14. 2016KeystoneML: Optimizing Pipelines for Large-Scale Advanced Analyticscited by 13
  15. 2018Serverless Computing: One Step Forward, Two Steps Backcited by 12
  16. 2014High-Performance Distributed ML at Scale through Parameter Server Consistency Modelscited by 10
  17. 2017Serving deep learning models in a serverless platformcited by 9
  18. 2018In-RDBMS Hardware Acceleration of Advanced Analyticscited by 8
  19. 2019OverSketched Newton: Fast Convex Optimization for Serverless Systemscited by 7
  20. 2019Lambada: Interactive Data Analytics on Cold Data using Serverless Cloud Infrastructurecited by 6
  21. 2019An Experimental Evaluation of Large Scale GBDT Systemscited by 5
  22. 2014An Asynchronous Distributed Proximal Gradient Method for Composite Convex Optimizationcited by 4
  23. 2017A Cost-based Optimizer for Gradient Descent Optimizationcited by 4
  24. 2018numpywren: serverless linear algebracited by 4
  25. 2019Starling: A Scalable Query Engine on Cloud Function Servicescited by 4
  26. 2020Dynamic Parameter Allocation in Parameter Serverscited by 4
  27. 2019Stratum: A Serverless Framework for Lifecycle Management of Machine Learning based Data Analytics Taskscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 3; 2023: 4; 2024: 2; 2025: 2; 2026: 3.

16 papers.

Sort:

  1. 2021FedLess: Secure and Scalable Federated Learning Using Serverless Computingcited by 13
  2. 2026Scheduling LLM Inference with Uncertainty-Aware Output Length Predictionscited by 8
  3. 2021You Do Not Need a Bigger Boat: Recommendations at Reasonable Scale in a (Mostly) Serverless and Open Stackcited by 5
  4. 2024PIM-Opt: Demystifying Distributed Optimization Algorithms on a Real-World Processing-In-Memory Systemcited by 5
  5. 2022SMLT: A Serverless Framework for Scalable and Adaptive Machine Learning Design and Trainingcited by 4
  6. 2023Exploring the Impact of Serverless Computing on Peer To Peer Training Machine Learningcited by 3
  7. 2023SPIRT: A Fault-Tolerant and Reliable Peer-to-Peer Serverless ML Training Architecturecited by 3
  8. 2022FedLesScan: Mitigating Stragglers in Serverless Federated Learningcited by 2
  9. 2024FSD-Inference: Fully Serverless Distributed Inference with Scalable Cloud Communicationcited by 2
  10. 2022MLProxy: SLA-Aware Reverse Proxy for Machine Learning Inference Serving on Serverless Computing Platformscited by 1
  11. 2023Architecting Peer-to-Peer Serverless Distributed Machine Learning Training for Improved Fault Tolerancecited by 0
  12. 2023Cloudless-Training: A Framework to Improve Efficiency of Geo-Distributed ML Trainingcited by 0
4 more
  1. 2025Scalable and Cost-Efficient ML Inference: Parallel Batch Processing with Serverless Functionscited by 0
  2. 2025FLStore: Efficient Federated Learning Storage for non-training workloadscited by 0
  3. 2026Word Frequency Counting Based on Serverless MapReducecited by 0
  4. 2026Shard the Gradient, Scale the Model: Serverless Federated Aggregation via Gradient Partitioningcited by 0