# Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud

arXiv: https://arxiv.org/abs/2411.15664 · HTML: https://mlsanity.com/paper/2411.15664/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 31 references that are in the map.
- First posted on arXiv 23 Nov 2024.

## Builds on (most cited first, 8 of 8)

- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://mlsanity.com/paper/2104.07857/) (2021)
- [Fast Distributed Inference Serving for Large Language Models](https://mlsanity.com/paper/2305.05920/) (2023)
- [ServerlessLLM: Low-Latency Serverless Inference for Large Language Models](https://mlsanity.com/paper/2401.14351/) (2024)
- [Cold Start Latency in Serverless Computing: A Systematic Review, Taxonomy, and Future Directions](https://mlsanity.com/paper/2310.08437/) (2023)
- [Rise of the Planet of Serverless Computing: A Systematic Review](https://mlsanity.com/paper/2206.12275/) (2022)
- [FaaSLight: General Application-Level Cold-Start Latency Optimization for Function-as-a-Service in Serverless Computing](https://mlsanity.com/paper/2207.08175/) (2022)
- [Mitigating Cold Starts in Serverless Platforms: A Pool-Based Approach](https://mlsanity.com/paper/1903.12221/) (2019)

## Built on by (most cited first, 2 of 2)

- [A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigms](https://mlsanity.com/paper/2504.16420/) (2025)
- [Addressing the sustainable AI trilemma: a case study on LLM agents and RAG](https://mlsanity.com/paper/2501.08262/) (2025)

## Abstract (arXiv)

This review report discusses the cold start latency in serverless inference and existing solutions. It particularly reviews the ServerlessLLM method, a system designed to address the cold start problem in serverless inference for large language models. Traditional serverless approaches struggle with high latency due to the size of LLM checkpoints and the overhead of initializing GPU resources. ServerlessLLM introduces a multitier checkpoint loading system, leveraging underutilized GPU memory and storage to reduce startup times by 6--8x compared to existing methods. It also proposes live inference migration and a startup-time-optimized model scheduler, ensuring efficient resource allocation and minimizing delays. This system significantly improves performance and scalability in serverless environments for LLM workloads. Besides ServerlessLLM, several other methods from recent research literature, including Rainbowcake, are reviewed in this paper. Further discussions explore how FaaS providers tackle cold starts and the possible future scopes.