mlsanityData to 14 Oct 2026

LUMEN: Coordinated Failure Recovery for Distributed LLM Serving

arXiv 2606.17787 ↗First posted 16 Jun 2026cs.DCcited by 1

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Modern large language model (LLM) serving clusters distribute inference requests across multiple worker processes on different GPUs, but failures are prevalent at scale. When a worker fails, the cluster simultaneously loses the failed worker's GPU-resident key-value (KV) caches and serving capacity, leaving surviving workers to absorb the redirected traffic while re-running interrupted requests from scratch. Existing fault-tolerant systems either restart interrupted requests from scratch or restore KV caches from checkpoints stored on a fixed neighboring worker, but both approaches route recovery work without considering current cluster load and leave the recovering worker idle during model reload. We present LUMEN, a fault-tolerant LLM serving system that treats recovery as a load-aware coordination problem across three decision points: checkpoint placement before failures, interrupted-request distribution at failure time, and serving capacity restoration during model reload. We evaluate LUMEN using both prototype experiments and large-scale simulations and demonstrate significant improvements in serving and recovery times.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 43 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

28 of 43 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving26 Mar
  2. 2026Towards Resiliency in Large Language Model Serving with KevlarFlow30 Jan
  3. 2025FailSafe: High-performance Resilient Serving18 Nov
  4. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  7. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  6. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  9. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  10. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  11. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  12. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
9 more
  1. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Lengthcited by 85
  2. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  3. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  4. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  5. 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
  6. 2024Revisiting Reliability in Large-Scale Machine Learning Research Clusterscited by 28
  7. 2024ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Developmentcited by 24
  8. 2024ReCycle: Resilient Training of Large DNNs using Pipeline Adaptationcited by 14
  9. 2024AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Accelerationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamocited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Concordia: JIT-Compiled Persistent-Kernel Checkpointing for Fault-Tolerant LLM Inference
  2. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
  3. 2024KVDirect: Distributed Disaggregated LLM Inference
  4. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  5. 2026A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints
  6. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  7. 2025RAPID-LLM: Resilience-Aware Performance analysis of Infrastructure for Distributed LLM Training and Inference ↗
  8. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching