mlsanityData to 14 Oct 2026

HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds

arXiv 2502.15524 ↗First posted 21 Feb 2025cs.DCcited by 14

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

With the proliferation of large language model (LLM) variants, developers are turning to serverless computing for cost-efficient LLM deployment. However, public cloud providers often struggle to provide performance guarantees for serverless LLM serving due to significant cold start latency caused by substantial model sizes and complex runtime dependencies. To address this problem, we present HydraServe, a serverless LLM serving system designed to minimize cold start latency in public clouds. HydraServe proactively distributes models across servers to quickly fetch them, and overlaps cold-start stages within workers to reduce startup latency. Additionally, HydraServe strategically places workers across GPUs to avoid network contention among cold-start instances. To minimize resource consumption during cold starts, HydraServe further introduces pipeline consolidation that can merge groups of workers into individual serving endpoints. Our comprehensive evaluations under diverse settings demonstrate that HydraServe reduces the cold start latency by 1.7×-- 4.7× and improves service level objective attainment by 1.43×--1.74× compared to baselines.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 69 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

25 of 69 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  2. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  3. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving2 Apr
  4. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  6. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
  7. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  11. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  12. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
6 more
  1. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80
  2. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 75
  3. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 66
  4. 2018Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Trainingcited by 28
  5. 2024PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculationcited by 14
  6. 2024GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelismcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 10.

14 papers.

Sort:

By area:

  1. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Servingcited by 6
  2. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Servingcited by 5
  3. 2026PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Servingcited by 3
  4. 2025Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computingcited by 2
  5. 2026Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Startcited by 2
  6. 2025ServerlessLoRA: Enabling Low-Latency Serverless Multi-LoRA Servingcited by 1
  7. 2025GraphMend: Code Transformations for Fixing Graph Breaks in PyTorch 2cited by 1
  8. 2026MoEless: Efficient MoE LLM Serving via Serverless Computingcited by 1
  9. 2026ARL-Tangram: Unleash the Resource Efficiency in Agentic Reinforcement Learningcited by 0
  10. 2026CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Servingcited by 0
  11. 2026TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructurecited by 0
  12. 2026Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamocited by 0
2 more
  1. 2026Stable Token-level Scheduling for Serverless LLM Inferencecited by 0
  2. 2026FastEEP: Fast Intra-Instance Elastic Scaling for Expert-Parallel MoE Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling
  2. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments
  3. 2025DeepServe: Serverless Large Language Model Serving at Scale
  4. 2026ExaServe: Large-Scale LLM Serving on Exascale HPC Systems
  5. 2025Tangram: Accelerating Serverless LLM Loading through GPU Memory Reuse and Affinity
  6. 2024Queue management for slo-oriented large language model serving
  7. 2026MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving
  8. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms