# DeepServe: Serverless Large Language Model Serving at Scale

arXiv: https://arxiv.org/abs/2501.14417 · HTML: https://mlsanity.com/paper/2501.14417/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jan 2025.

## Built on by (most cited first, 18 of 18)

- [TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity](https://mlsanity.com/paper/2512.03416/) (2025)
- [LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure](https://mlsanity.com/paper/2602.23036/) (2026)
- [WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving](https://mlsanity.com/paper/2512.09472/) (2025)
- [Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stage](https://mlsanity.com/paper/2601.03043/) (2026)
- [Making MoE-based LLM Inference Resilient with Tarragon](https://mlsanity.com/paper/2601.01310/) (2026)
- [Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start](https://mlsanity.com/paper/2604.06664/) (2026)
- [SwiftCache: Efficient LLM Serving for Multi-turn Conversations with Heterogeneous KV Cache Sharing](https://mlsanity.com/paper/2606.16135/) (2026)
- [MoEless: Efficient MoE LLM Serving via Serverless Computing](https://mlsanity.com/paper/2603.06350/) (2026)
- [ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse](https://mlsanity.com/paper/2605.22850/) (2026)
- [DiLaServe: High SLO Attainment Serving for Diffusion Language Models](https://mlsanity.com/paper/2606.29094/) (2026)
- [NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decoding](https://mlsanity.com/paper/2605.21100/) (2026)
- [HyperParallel-MoE: Multi-Core Interleaved Scheduling for Fast MoE Training on Ascend NPUs](https://mlsanity.com/paper/2605.23764/) (2026)
- [Breaking the Ice: Analyzing Cold Start Latency in vLLM](https://mlsanity.com/paper/2606.07362/) (2026)
- [Communication-Efficient Verifiable Attention for LLM Inference](https://mlsanity.com/paper/2606.16352/) (2026)
- [ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill](https://mlsanity.com/paper/2606.22541/) (2026)
- [ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platform](https://mlsanity.com/paper/2607.26566/) (2026)
- [OpScale: Operator-level Provisioning and Autoscaling for LLM Serving](https://mlsanity.com/paper/2608.13499/) (2026)
- [Stable Token-level Scheduling for Serverless LLM Inference](https://mlsanity.com/paper/2610.17689/) (2026)

## Abstract (arXiv)

In this paper, we propose DEEPSERVE, a scalable and serverless AI platform designed to efficiently serve large language models (LLMs) at scale in cloud environments. DEEPSERVE addresses key challenges such as resource allocation, serving efficiency, and cold start latencies through four main design components. First, DEEPSERVE uses a simple serverless abstraction called the request-job-task model, which helps manage diverse AI workloads across posttraining and model-serving tasks. Second, DEEPSERVE integrates an in-house serving engine named FLOWSERVE using a microkernel-inspired design, NPU-centric execution, and SPMD-based parallelism to optimize LLM serving. Third, DEEPSERVE includes novel scheduling policies tailored for a configuration with both PD-disaggregated and PD-colocated instances. Fourth, DEEPSERVE includes optimizations such as pre-warmed pods, DRAM pre-loading, and NPU-fork, which allow DEEPSERVE to scale up to 64 instances in seconds. DEEPSERVE has been in production for over a year, operating on a large Ascend NPU cluster and providing industrystandard APIs for fine-tuning, agent serving, and model serving to our customers.