# BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Caching

arXiv: https://arxiv.org/abs/2412.17246 · HTML: https://mlsanity.com/paper/2412.17246/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2024.

## Built on by (most cited first, 25 of 25)

- [RollArt: Disaggregated Multi-Task Agentic RL Training at Scale](https://mlsanity.com/paper/2512.22560/) (2025)
- [Slim-SC: Thought Pruning for Efficient Scaling with Self-Consistency](https://mlsanity.com/paper/2509.13990/) (2025)
- [TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity](https://mlsanity.com/paper/2512.03416/) (2025)
- [MSCCL++: Rethinking GPU Communication Abstractions for AI Inference](https://mlsanity.com/paper/2504.09014/) (2025)
- [Janus: Disaggregating Attention and Experts for Scalable MoE Inference](https://mlsanity.com/paper/2512.13525/) (2025)
- [FairBatching: Fairness-Aware Batch Formation for LLM Inference](https://mlsanity.com/paper/2510.14392/) (2025)
- [ElasticMoE: An Efficient Auto Scaling Method for Mixture-of-Experts Models](https://mlsanity.com/paper/2510.02613/) (2025)
- [BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure](https://mlsanity.com/paper/2510.13223/) (2025)
- [Agentic Witnessing: Pragmatic and Scalable TEE-Enabled Privacy-Preserving Auditing](https://mlsanity.com/paper/2604.24203/) (2026)
- [SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models](https://mlsanity.com/paper/2505.07680/) (2025)
- [Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start](https://mlsanity.com/paper/2604.06664/) (2026)
- [Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving](https://mlsanity.com/paper/2606.01839/) (2026)
- [SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling](https://mlsanity.com/paper/2607.08565/) (2026)
- [SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer](https://mlsanity.com/paper/2607.28150/) (2026)
- [ServerlessLoRA: Enabling Low-Latency Serverless Multi-LoRA Serving](https://mlsanity.com/paper/2505.14468/) (2025)
- [Multi-Layer Scheduling for MoE-Based LLM Reasoning](https://mlsanity.com/paper/2602.21626/) (2026)
- [Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines](https://mlsanity.com/paper/2604.15186/) (2026)
- [ARL-Tangram: Unleash the Resource Efficiency in Agentic Reinforcement Learning](https://mlsanity.com/paper/2603.13019/) (2026)
- [CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving](https://mlsanity.com/paper/2607.04181/) (2026)
- [Auto-Scaling Heterogeneous Neural Processing Units for Energy and Cost-Efficient LLM Serving](https://mlsanity.com/paper/2607.16488/) (2026)
- [TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure](https://mlsanity.com/paper/2608.06007/) (2026)
- [Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamo](https://mlsanity.com/paper/2609.25451/) (2026)
- [Cross-Model Autoscaling for Shared LLM Serving](https://mlsanity.com/paper/2609.29160/) (2026)
- [FastEEP: Fast Intra-Instance Elastic Scaling for Expert-Parallel MoE Serving](https://mlsanity.com/paper/2610.18451/) (2026)
- [JointServe: Serving Multi-Model Agents on Memory-Constrained GPUs](https://mlsanity.com/paper/2610.19822/) (2026)

## Abstract (arXiv)

Model autoscaling is the key mechanism to achieve serverless model-as-a-service, but it faces a fundamental trade-off between scaling speed and storage/memory usage to cache parameters, and cannot meet frequent scaling requirements across multiple hosts. The key problem is that data plane performance is slow, and scaled instances remain stopped while parameters are loading. In this paper, we first show that the data plane can be made fast with no or O(1) caching by loading parameters through the compute network between GPUs because: (1) its speed is comparable to host cache and is underutilized, and (2) scaling multiple instances requires no or O(1) caching with network-optimized multicast. Second, autoscaling can be made live by breaking the scaling abstraction for inference from a coarse-grained instance-level to a fine-grained layer-level. This allows us to offload the layer computation from the overloaded serving instances to the scaled ones without waiting for the parameters to be fully loaded. Under real-world workloads, our system BLITZSCALE achieves up to 94 % lower tail latency reductions compared to state-of-the-art autoscaling system (ServerlessLLM), and it reduces the GPU time used for serving by 49 % when compared with serving systems that do not support autoscaling like DistServe and vLLM with the same service-level-agreement.