# Towards Resiliency in Large Language Model Serving with KevlarFlow

arXiv: https://arxiv.org/abs/2601.22438 · HTML: https://mlsanity.com/paper/2601.22438/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 27 of its 57 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 27 of 27)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [Llumnix: Dynamic Scheduling for Large Language Model Serving](https://mlsanity.com/paper/2406.03243/) (2024)
- [The Hardware Lottery](https://mlsanity.com/paper/2009.06489/) (2020)
- [SpotServe: Serving Generative Large Language Models on Preemptible Instances](https://mlsanity.com/paper/2311.15566/) (2023)
- [BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems](https://mlsanity.com/paper/2401.17644/) (2024)
- [D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving](https://mlsanity.com/paper/2403.01876/) (2024)
- [Queue management for slo-oriented large language model serving](https://mlsanity.com/paper/2407.00047/) (2024)
- [ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments](https://mlsanity.com/paper/2502.09334/) (2025)
- [Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithms](https://mlsanity.com/paper/2507.04786/) (2025)
- [Past-Future Scheduler for LLM Serving under SLA Guarantees](https://mlsanity.com/paper/2507.10150/) (2025)
- [Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instances](https://mlsanity.com/paper/2403.14097/) (2024)
- [ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation](https://mlsanity.com/paper/2405.14009/) (2024)
- [Hierarchical Autoscaling for Large Language Model Serving with Chiron](https://mlsanity.com/paper/2501.08090/) (2025)
- [TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inference](https://mlsanity.com/paper/2506.10470/) (2025)
- [ENOVA: Autoscaling towards Cost-effective and Stable Serverless LLM Serving](https://mlsanity.com/paper/2407.09486/) (2024)
- [Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving](https://mlsanity.com/paper/2504.07494/) (2025)
- [gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling](https://mlsanity.com/paper/2504.14775/) (2025)
- [Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs](https://mlsanity.com/paper/2503.11901/) (2025)
- [Reliable and Resilient Collective Communication Library for LLM Training and Serving](https://mlsanity.com/paper/2512.25059/) (2025)
- [AnchorTP: Resilient LLM Inference with State-Preserving Elastic Tensor Parallelism](https://mlsanity.com/paper/2511.11617/) (2025)
- [ALISE: Accelerating Large Language Model Serving with Speculative Scheduling](https://mlsanity.com/paper/2410.23537/) (2024)
- [An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models](https://mlsanity.com/paper/2501.12469/) (2025)

## Built on by (most cited first, 3 of 3)

- [LUMEN: Coordinated Failure Recovery for Distributed LLM Serving](https://mlsanity.com/paper/2606.17787/) (2026)
- [FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models](https://mlsanity.com/paper/2606.27866/) (2026)
- [Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamo](https://mlsanity.com/paper/2609.25451/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) serving systems remain fundamentally fragile, where frequent hardware faults in hyperscale clusters trigger disproportionate service outages in the software stack. Current recovery mechanisms are prohibitively slow, often requiring up to 10 minutes to reinitialize resources and reload massive model weights. We introduce KevlarFlow, a fault tolerant serving architecture designed to bridge the gap between hardware unreliability and service availability. KevlarFlow leverages 1) decoupled model parallelism initialization, 2) dynamic traffic rerouting, and 3) background KV cache replication to maintain high throughput during partial failures. Our evaluation demonstrates that KevlarFlow reduces mean-time-to-recovery (MTTR) by 20x and, under failure conditions, improves average latency by 3.1x, 99th percentile (p99) latency by 2.8x, average time-to-first-token (TTFT) by 378.9x, and p99 TTFT by 574.6x with negligible runtime overhead in comparison to state-of-the-art LLM serving systems.