mlsanityData to 13 Oct 2026

Towards Resiliency in Large Language Model Serving with KevlarFlow

arXiv 2601.22438 ↗First posted 30 Jan 2026cs.DC cs.CL cs.LGcited by 3

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large Language Model (LLM) serving systems remain fundamentally fragile, where frequent hardware faults in hyperscale clusters trigger disproportionate service outages in the software stack. Current recovery mechanisms are prohibitively slow, often requiring up to 10 minutes to reinitialize resources and reload massive model weights. We introduce KevlarFlow, a fault tolerant serving architecture designed to bridge the gap between hardware unreliability and service availability. KevlarFlow leverages 1) decoupled model parallelism initialization, 2) dynamic traffic rerouting, and 3) background KV cache replication to maintain high throughput during partial failures. Our evaluation demonstrates that KevlarFlow reduces mean-time-to-recovery (MTTR) by 20x and, under failure conditions, improves average latency by 3.1x, 99th percentile (p99) latency by 2.8x, average time-to-first-token (TTFT) by 378.9x, and p99 TTFT by 574.6x with negligible runtime overhead in comparison to state-of-the-art LLM serving systems.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 57 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

27 of 57 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Past-Future Scheduler for LLM Serving under SLA Guarantees14 Jul
  2. 2025gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling21 Apr
  3. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving10 Apr
  4. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments13 Feb
  5. 2024ALISE: Accelerating Large Language Model Serving with Speculative Scheduling31 Oct
  6. 2024Queue management for slo-oriented large language model serving5 Jun
  7. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  9. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  10. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  11. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov

Other references · 16

  1. 2024The Llama 3 Herd of Modelscited by 14,486
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,831
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,851
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 626
  5. 2020The Hardware Lotterycited by 160
  6. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  7. 2025Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithmscited by 19
  8. 2024Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instancescited by 17
  9. 2024ReCycle: Resilient Training of Large DNNs using Pipeline Adaptationcited by 14
  10. 2025Hierarchical Autoscaling for Large Language Model Serving with Chironcited by 9
  11. 2025TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inferencecited by 8
  12. 2024ENOVA: Autoscaling towards Cost-effective and Stable Serverless LLM Servingcited by 6
4 more
  1. 2025Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUscited by 4
  2. 2025Reliable and Resilient Collective Communication Library for LLM Training and Servingcited by 4
  3. 2025AnchorTP: Resilient LLM Inference with State-Preserving Elastic Tensor Parallelismcited by 3
  4. 2025An Empirical Characterization of Outages and Incidents in Public Services for Large Language Modelscited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026LUMEN: Coordinated Failure Recovery for Distributed LLM Servingcited by 1
  2. 2026FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Modelscited by 0
  3. 2026Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamocited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving
  2. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
  3. 2025FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters
  4. 2025FailSafe: High-performance Resilient Serving
  5. 2026Characterizing Contention-Induced Reliability Collapse in KV-Cache Timing Side Channels for Multi-Tenant LLM Serving
  6. 2025BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure
  7. 2026ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving
  8. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing