mlsanityData to 14 Oct 2026

ServerlessLLM: Low-Latency Serverless Inference for Large Language Models

arXiv 2401.14351 ↗First posted 25 Jan 2024cs.LG cs.DCcited by 80

Not on the map. Why?

Cited by 39 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents ServerlessLLM, a distributed system designed to support low-latency serverless inference for Large Language Models (LLMs). By harnessing the substantial near-GPU storage and memory capacities of inference servers, ServerlessLLM achieves effective local checkpoint storage, minimizing the need for remote checkpoint downloads and ensuring efficient checkpoint loading. The design of ServerlessLLM features three core contributions: (i) fast multi-tier checkpoint loading, featuring a new loading-optimized checkpoint format and a multi-tier loading system, fully utilizing the bandwidth of complex storage hierarchies on GPU servers; (ii) efficient live migration of LLM inference, which enables newly initiated inferences to capitalize on local checkpoint storage while ensuring minimal user interruption; and (iii) startup-time-optimized model scheduling, which assesses the locality statuses of checkpoints on each server and schedules the model onto servers that minimize the time to start the inference. Comprehensive evaluations, including microbenchmarks and real-world scenarios, demonstrate that ServerlessLLM dramatically outperforms state-of-the-art serverless systems, reducing latency by 10 - 200X across various LLM inference workloads.

Key facts. Cited by 80 papers in the map (data through 14 Oct 2026). Builds on 19 of its 83 references that are in the map. First posted on arXiv 25 Jan 2024.

Builds on

19 of 83 references.

References · 19

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  7. 2022Efficiently Scaling Transformer Inferencecited by 539
  8. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  9. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  10. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  11. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  12. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 66
7 more
  1. 2023Torpor: GPU-Enabled Serverless Computing for Low-Latency, Resource-Efficient Inferencecited by 13
  2. 2021Benchmarking, Analysis, and Optimization of Serverless Function Snapshotscited by 12
  3. 2022No Provisioned Concurrency: Fast RDMA-codesigned Remote Fork for Serverless Computingcited by 8
  4. 2020Faasm: Lightweight Isolation for Efficient Stateful Serverless Computingcited by 7
  5. 2021FaaSNet: Scalable and Fast Provisioning of Custom Serverless Container Runtimes at Alibaba Cloud Function Computecited by 7
  6. 2021Following the Data, Not the Function: Rethinking Function Orchestration in Serverless Computingcited by 7
  7. 2021FaaT: A Transparent Auto-Scaling Cache for Serverless Applicationscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 40; 2026: 33.

80 papers.

Sort:

By area:

  1. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 32
  2. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  3. 2024ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Developmentcited by 24
  4. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooningcited by 24
  5. 2025Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inferencecited by 22
  6. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  7. 2025A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Servicescited by 17
  8. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Cloudscited by 14
  9. 2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scalecited by 14
  10. 2025Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptationcited by 13
  11. 2025Slim-SC: Thought Pruning for Efficient Scaling with Self-Consistencycited by 13
  12. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 12
38 more
  1. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Servingcited by 11
  2. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  3. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  4. 2025ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictorcited by 9
  5. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Managementcited by 8
  6. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  7. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inferencecited by 8
  8. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compressioncited by 8
  9. 2024Opportunities and Challenges of Generative-AI in Financecited by 6
  10. 2025MoE2: Optimizing Collaborative Inference for Edge Large Language Modelscited by 6
  11. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizingcited by 6
  12. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Servingcited by 6
  13. 2026SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusterscited by 6
  14. 2026SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inferencecited by 6
  15. 2024ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Servingcited by 5
  16. 2025PolyServe: Efficient Multi-SLO Serving at Scalecited by 5
  17. 2025Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokenscited by 5
  18. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Servingcited by 5
  19. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  20. 2025BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMscited by 4
  21. 2025Fast LLM Post-training via Decoupled and Fastest-of-N Speculationcited by 4
  22. 2025Adaptive GPU Resource Allocation for Multi-Agent Collaborative Reasoning in Serverless Environmentscited by 4
  23. 2026PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Servingcited by 4
  24. 2025Optimizing Distributed Deployment of Mixture-of-Experts Model Inference in Serverless Computingcited by 3
  25. 2025FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Frameworkcited by 3
  26. 2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systemscited by 3
  27. 2025Predictive-LoRA: A Proactive and Fragmentation-Aware Serverless Inference System for LLMscited by 3
  28. 2025A Plan Reuse Mechanism for LLM-Driven Agentcited by 3
  29. 2026Agentic Witnessing: Pragmatic and Scalable TEE-Enabled Privacy-Preserving Auditingcited by 3
  30. 2024Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloudcited by 2
  31. 2025Recursive Offloading for LLM Serving in Multi-tier Networkscited by 2
  32. 2025MACE: A Hybrid LLM Serving System with Colocated SLO-aware Continuous Retraining Alignmentcited by 2
  33. 2025Beyond Training: Enabling Self-Evolution of Agents with MOBIMEMcited by 2
  34. 2026Potential Applications of HBF in LLM Serving Systemscited by 2
  35. 2025PlanetServe: A Decentralized, Scalable, and Privacy-Preserving Overlay for Democratizing Large Language Model Servingcited by 1
  36. 2025HybridServe: Efficient Serving of Large AI Models with Confidence-Based Cascade Routingcited by 1
  37. 2025ServerlessLoRA: Enabling Low-Latency Serverless Multi-LoRA Servingcited by 1
  38. 2025TaleFrame: An Interactive Story Generation System with Fine-Grained Control and Large Language Modelscited by 1

50 most cited of 80 All 80 (JSON)