# Towards Demystifying Serverless Machine Learning Training

arXiv: https://arxiv.org/abs/2105.07806 · HTML: https://mlsanity.com/paper/2105.07806/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 104 references that are in the map.
- First posted on arXiv 17 May 2021.

## Builds on (most cited first, 30 of 39)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent](https://mlsanity.com/paper/1705.09056/) (2017)
- [TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning](https://mlsanity.com/paper/1705.07878/) (2017)
- [The Convergence of Sparsified Gradient Methods](https://mlsanity.com/paper/1809.10505/) (2018)
- [Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication](https://mlsanity.com/paper/1902.00340/) (2019)
- [Don't Use Large Mini-Batches, Use Local SGD](https://mlsanity.com/paper/1808.07217/) (2018)
- [On the Convergence of Decentralized Gradient Descent](https://mlsanity.com/paper/1310.7063/) (2013)
- [QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding](https://mlsanity.com/paper/1610.02132/) (2016)
- [D2: Decentralized Training over Decentralized Data](https://mlsanity.com/paper/1803.07068/) (2018)
- [ATOMO: Communication-efficient Learning via Atomic Sparsification](https://mlsanity.com/paper/1806.04090/) (2018)
- [Asynchronous Stochastic Gradient Descent with Delay Compensation](https://mlsanity.com/paper/1609.08326/) (2016)
- [Communication Compression for Decentralized Training](https://mlsanity.com/paper/1803.06443/) (2018)
- [Byzantine Stochastic Gradient Descent](https://mlsanity.com/paper/1803.08917/) (2018)
- [Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD](https://mlsanity.com/paper/1810.08313/) (2018)
- [Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization](https://mlsanity.com/paper/1806.08054/) (2018)
- [CoCoA: A General Framework for Communication-Efficient Distributed Optimization](https://mlsanity.com/paper/1611.02189/) (2016)
- [MLlib: Machine Learning in Apache Spark](https://mlsanity.com/paper/1505.06807/) (2015)
- [COLA: Decentralized Linear Learning](https://mlsanity.com/paper/1808.04883/) (2018)
- [TicTac: Accelerating Distributed Deep Learning with Communication Scheduling](https://mlsanity.com/paper/1803.03288/) (2018)
- [Straggler Mitigation in Distributed Optimization Through Data Encoding](https://mlsanity.com/paper/1711.04969/) (2017)
- [Gossip Dual Averaging for Decentralized Optimization of Pairwise Functions](https://mlsanity.com/paper/1606.02421/) (2016)
- [Adding vs. Averaging in Distributed Primal-Dual Optimization](https://mlsanity.com/paper/1502.03508/) (2015)
- [Asynchronous stochastic convex optimization](https://mlsanity.com/paper/1508.00882/) (2015)
- [Serverless Computing: Current Trends and Open Problems](https://mlsanity.com/paper/1706.03178/) (2017)
- [Distributed Learning Systems with First-order Methods](https://mlsanity.com/paper/2104.05245/) (2021)
- [KeystoneML: Optimizing Pipelines for Large-Scale Advanced Analytics](https://mlsanity.com/paper/1610.09451/) (2016)
- [Serverless Computing: One Step Forward, Two Steps Back](https://mlsanity.com/paper/1812.03651/) (2018)
- [High-Performance Distributed ML at Scale through Parameter Server Consistency Models](https://mlsanity.com/paper/1410.8043/) (2014)
- [Serving deep learning models in a serverless platform](https://mlsanity.com/paper/1710.08460/) (2017)
- [In-RDBMS Hardware Acceleration of Advanced Analytics](https://mlsanity.com/paper/1801.06027/) (2018)

## Built on by (most cited first, 16 of 16)

- [FedLess: Secure and Scalable Federated Learning Using Serverless Computing](https://mlsanity.com/paper/2111.03396/) (2021)
- [Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions](https://mlsanity.com/paper/2604.00499/) (2026)
- [You Do Not Need a Bigger Boat: Recommendations at Reasonable Scale in a (Mostly) Serverless and Open Stack](https://mlsanity.com/paper/2107.07346/) (2021)
- [PIM-Opt: Demystifying Distributed Optimization Algorithms on a Real-World Processing-In-Memory System](https://mlsanity.com/paper/2404.07164/) (2024)
- [SMLT: A Serverless Framework for Scalable and Adaptive Machine Learning Design and Training](https://mlsanity.com/paper/2205.01853/) (2022)
- [Exploring the Impact of Serverless Computing on Peer To Peer Training Machine Learning](https://mlsanity.com/paper/2309.14139/) (2023)
- [SPIRT: A Fault-Tolerant and Reliable Peer-to-Peer Serverless ML Training Architecture](https://mlsanity.com/paper/2309.14148/) (2023)
- [FedLesScan: Mitigating Stragglers in Serverless Federated Learning](https://mlsanity.com/paper/2211.05739/) (2022)
- [FSD-Inference: Fully Serverless Distributed Inference with Scalable Cloud Communication](https://mlsanity.com/paper/2403.15195/) (2024)
- [MLProxy: SLA-Aware Reverse Proxy for Machine Learning Inference Serving on Serverless Computing Platforms](https://mlsanity.com/paper/2202.11243/) (2022)
- [Architecting Peer-to-Peer Serverless Distributed Machine Learning Training for Improved Fault Tolerance](https://mlsanity.com/paper/2302.13995/) (2023)
- [Cloudless-Training: A Framework to Improve Efficiency of Geo-Distributed ML Training](https://mlsanity.com/paper/2303.05330/) (2023)
- [Scalable and Cost-Efficient ML Inference: Parallel Batch Processing with Serverless Functions](https://mlsanity.com/paper/2502.12017/) (2025)
- [FLStore: Efficient Federated Learning Storage for non-training workloads](https://mlsanity.com/paper/2503.00323/) (2025)
- [Word Frequency Counting Based on Serverless MapReduce](https://mlsanity.com/paper/2601.00380/) (2026)
- [Shard the Gradient, Scale the Model: Serverless Federated Aggregation via Gradient Partitioning](https://mlsanity.com/paper/2604.22072/) (2026)

## Abstract (arXiv)

The appeal of serverless (FaaS) has triggered a growing interest on how to use it in data-intensive applications such as ETL, query processing, or machine learning (ML). Several systems exist for training large-scale ML models on top of serverless infrastructures (e.g., AWS Lambda) but with inconclusive results in terms of their performance and relative advantage over "serverful" infrastructures (IaaS). In this paper we present a systematic, comparative study of distributed ML training over FaaS and IaaS. We present a design space covering design choices such as optimization algorithms and synchronization protocols, and implement a platform, LambdaML, that enables a fair comparison between FaaS and IaaS. We present experimental results using LambdaML, and further develop an analytic model to capture cost/performance tradeoffs that must be considered when opting for a serverless infrastructure. Our results indicate that ML training pays off in serverless only for models with efficient (i.e., reduced) communication and that quickly converge. In general, FaaS can be much faster but it is never significantly cheaper than IaaS.