# FSD-Inference: Fully Serverless Distributed Inference with Scalable Cloud Communication

arXiv: https://arxiv.org/abs/2403.15195 · HTML: https://mlsanity.com/paper/2403.15195/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 63 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Builds on (most cited first, 16 of 16)

- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [A Survey on Distributed Machine Learning](https://mlsanity.com/paper/1912.09789/) (2019)
- [Occupy the Cloud: Distributed Computing for the 99%](https://mlsanity.com/paper/1702.04024/) (2017)
- [Towards Demystifying Serverless Machine Learning Training](https://mlsanity.com/paper/2105.07806/) (2021)
- [Serverless Computing: A Survey of Opportunities, Challenges and Applications](https://mlsanity.com/paper/1911.01296/) (2019)
- [Serving deep learning models in a serverless platform](https://mlsanity.com/paper/1710.08460/) (2017)
- [Sparse Deep Neural Network Graph Challenge](https://mlsanity.com/paper/1909.05631/) (2019)
- [Lambada: Interactive Data Analytics on Cold Data using Serverless Cloud Infrastructure](https://mlsanity.com/paper/1912.00937/) (2019)
- [The Serverless Computing Survey: A Technical Primer for Design Architecture](https://mlsanity.com/paper/2112.12921/) (2021)
- [Wukong: A Scalable and Locality-Enhanced Framework for Serverless Parallel Computing](https://mlsanity.com/paper/2010.07268/) (2020)
- [Starling: A Scalable Query Engine on Cloud Function Services](https://mlsanity.com/paper/1911.11727/) (2019)
- [Partitioning sparse deep neural networks for scalable training and inference](https://mlsanity.com/paper/2104.11805/) (2021)
- [Serverless Data Science -- Are We There Yet? A Case Study of Model Serving](https://mlsanity.com/paper/2103.02958/) (2021)
- [On-demand Container Loading in AWS Lambda](https://mlsanity.com/paper/2305.13162/) (2023)
- [In Search of a Fast and Efficient Serverless DAG Engine](https://mlsanity.com/paper/1910.05896/) (2019)
- [Reliable Transactions in Serverless-Edge Architecture](https://mlsanity.com/paper/2201.00982/) (2022)

## Built on by (most cited first, 2 of 2)

- [SQUASH: Serverless and Distributed Quantization-based Attributed Vector Similarity Search](https://mlsanity.com/paper/2502.01528/) (2025)
- [High-Performance Serverless Computing: A Systematic Literature Review on Serverless for HPC, AI, and Big Data](https://mlsanity.com/paper/2601.09334/) (2026)

## Abstract (arXiv)

Serverless computing offers attractive scalability, elasticity and cost-effectiveness. However, constraints on memory, CPU and function runtime have hindered its adoption for data-intensive applications and machine learning (ML) workloads. Traditional 'server-ful' platforms enable distributed computation via fast networks and well-established inter-process communication (IPC) mechanisms such as MPI and shared memory. In the absence of such solutions in the serverless domain, parallel computation with significant IPC requirements is challenging. We present FSD-Inference, the first fully serverless and highly scalable system for distributed ML inference. We explore potential communication channels, in conjunction with Function-as-a-Service (FaaS) compute, to design a state-of-the-art solution for distributed ML within the context of serverless data-intensive computing. We introduce novel fully serverless communication schemes for ML inference workloads, leveraging both cloud-based publish-subscribe/queueing and object storage offerings. We demonstrate how publish-subscribe/queueing services can be adapted for FaaS IPC with comparable performance to object storage, while offering significantly reduced cost at high parallelism levels. We conduct in-depth experiments on benchmark DNNs of various sizes. The results show that when compared to server-based alternatives, FSD-Inference is significantly more cost-effective and scalable, and can even achieve competitive performance against optimized HPC solutions. Experiments also confirm that our serverless solution can handle large distributed workloads and leverage high degrees of FaaS parallelism.