# λScale: Enabling Fast Scaling for Serverless Large Language Model Inference

arXiv: https://arxiv.org/abs/2502.09922 · HTML: https://mlsanity.com/paper/2502.09922/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2025.

## Built on by (most cited first, 8 of 8)

- [TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity](https://mlsanity.com/paper/2512.03416/) (2025)
- [Janus: Disaggregating Attention and Experts for Scalable MoE Inference](https://mlsanity.com/paper/2512.13525/) (2025)
- [PolyServe: Efficient Multi-SLO Serving at Scale](https://mlsanity.com/paper/2507.17769/) (2025)
- [ElasticMoE: An Efficient Auto Scaling Method for Mixture-of-Experts Models](https://mlsanity.com/paper/2510.02613/) (2025)
- [Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start](https://mlsanity.com/paper/2604.06664/) (2026)
- [ARL-Tangram: Unleash the Resource Efficiency in Agentic Reinforcement Learning](https://mlsanity.com/paper/2603.13019/) (2026)
- [The Model Parking Tax: Quantifying the Hidden Energy Cost of Always-On GPU Model Deployment](https://mlsanity.com/paper/2605.23918/) (2026)
- [Stable Token-level Scheduling for Serverless LLM Inference](https://mlsanity.com/paper/2610.17689/) (2026)

## Abstract (arXiv)

Serverless computing has emerged as a compelling solution for cloud-based model inference. However, as modern large language models (LLMs) continue to grow in size, existing serverless platforms often face substantial model startup overhead. This poses a significant challenge in efficiently scaling model instances to accommodate dynamic, bursty workloads commonly observed in real-world inference services. In this paper, we introduce λScale, an efficient serverless inference system to achieve fast model scaling. The key idea behind λScale is to leverage high-speed RDMA networks between GPU nodes for fast model multicast, while enabling distributed inference execution during model transmission -- referred to as "execute-while-load". λScale proposes an efficient model scaling scheme, λPipe, which supports adaptive model multicast and dynamically constructs execution pipelines across receiving nodes for collaborative, distributed inference. Additionally, λScale supports efficient model management across GPU and host memory, allowing fast scaling for models across different storage tiers. Evaluation results show that λScale enables fast model scaling and effectively handles load spikes, achieving up to 5x tail-latency improvement and 31.3% cost reduction compared to state-of-the-art solutions on real-world LLM inference traces.