mlsanityData to 14 Oct 2026

GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving

arXiv 2605.00831 ↗First posted 26 Mar 2026cs.DC cs.AI cs.PFcited by 3

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The rise of million-token, agent-based applications has placed unprecedented demands on large language model (LLM) inference services. The long-running nature of these tasks increases their susceptibility to hardware and software faults, leading to costly job failures, wasted resources, and degraded user experience. The stateful key-value (KV) cache, which grows with the sequence length, presents a central challenge as it is a critical and vulnerable component in distributed serving systems. In this work, we propose GhostServe, a novel checkpointing solution to facilitate fault-tolerant LLM serving. Specifically, GhostServe protects the streaming KV cache in the shadow by applying erasure coding to generate and store the parity shards in host memory. In the event of device failures, GhostServe enables fast reconstruction of the lost KV cache, allowing the inference process to resume seamlessly without costly full recomputation or state replication. Evaluations demonstrate that GhostServe reduces checkpointing latency by up to 2.7x and recovery latency by 2.1x for a single batch, and 1.2x median response latency compared to existing methods, in the presence of system failures, paving the way for high-availability and cost-effective LLM serving at scale.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 35 of its 64 references that are in the map. First posted on arXiv 26 Mar 2026.

Builds on

35 of 64 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs2 Feb
  2. 2024ALISE: Accelerating Large Language Model Serving with Speculative Scheduling31 Oct
  3. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  5. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
  6. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  11. 2022Efficiently Scaling Transformer Inferencecited by 539
  12. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
17 more
  1. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  2. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  3. 2024Revisiting Reliability in Large-Scale Machine Learning Research Clusterscited by 28
  4. 2024Context Parallelism for Scalable Million-Token Inferencecited by 26
  5. 2024ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Developmentcited by 24
  6. 2025Robust LLM Training Infrastructure at ByteDancecited by 16
  7. 2025Understanding Silent Data Corruption in LLM Trainingcited by 15
  8. 2025Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Expertscited by 15
  9. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 13
  10. 2025Understanding Stragglers in Large Model Training Using What-if Analysiscited by 12
  11. 2025LithOS: An Operating System for Efficient Machine Learning on GPUscited by 9
  12. 2026Training LLMs with Fault Tolerant HSDP on 100,000 GPUscited by 6
  13. 2025Enhancing reliability in AI inference services: An empirical study on real production incidentscited by 5
  14. 2025Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUscited by 4
  15. 2025Silent Data Corruption by 10x Test Escapes Threatens Reliable Computingcited by 3
  16. 2013CUDA Leaks: Information Leakage in GPU Architecturescited by 2
  17. 2017Erasure Coding for Small Objects in In-Memory KV Storagecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026LUMEN: Coordinated Failure Recovery for Distributed LLM Servingcited by 1
  2. 2026Belayer: Efficient Fault Tolerance for LLM Agentic RL Trainingcited by 0
  3. 2026Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamocited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Concordia: JIT-Compiled Persistent-Kernel Checkpointing for Fault-Tolerant LLM Inference
  2. 2026Continuous Discovery of Vulnerabilities in LLM Serving Systems with Fuzzing
  3. 2025MIST: A Co-Design Framework for Heterogeneous, Multi-Stage LLM Inference
  4. 2026Characterizing Contention-Induced Reliability Collapse in KV-Cache Timing Side Channels for Multi-Tenant LLM Serving
  5. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
  6. 2026The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems
  7. 2026Towards Resiliency in Large Language Model Serving with KevlarFlow
  8. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching