# Making MoE-based LLM Inference Resilient with Tarragon

arXiv: https://arxiv.org/abs/2601.01310 · HTML: https://mlsanity.com/paper/2601.01310/

## Key facts

- Cited by 2 papers in the map (data through 7 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 20 of its 46 references that are in the map.
- First posted on arXiv 4 Jan 2026.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 20 of 20)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding](https://mlsanity.com/paper/2507.19427/) (2025)
- [DeepServe: Serverless Large Language Model Serving at Scale](https://mlsanity.com/paper/2501.14417/) (2025)
- [Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving](https://mlsanity.com/paper/2509.17863/) (2025)
- [Lazarus: Resilient and Elastic Training of Mixture-of-Experts Models](https://mlsanity.com/paper/2407.04656/) (2024)
- [Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Scheduling](https://mlsanity.com/paper/2503.04398/) (2025)
- [InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceivers](https://mlsanity.com/paper/2502.03885/) (2025)
- [HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUs](https://mlsanity.com/paper/2504.03871/) (2025)
- [Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs](https://mlsanity.com/paper/2503.11901/) (2025)

## Built on by (most cited first, 2 of 2)

- [Crab: A Semantics-Aware Checkpoint/Restore Runtime for Agent Sandboxes](https://mlsanity.com/paper/2604.28138/) (2026)
- [Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamo](https://mlsanity.com/paper/2609.25451/) (2026)

## Abstract (arXiv)

Mixture-of-Experts (MoE) models are increasingly used to serve LLMs at scale, but failures become common as deployment scale grows. Existing systems exhibit poor failure resilience: even a single worker failure triggers a coarse-grained, service-wide restart, discarding accumulated progress and halting the entire inference pipeline during recovery--an approach clearly ill-suited for latency-sensitive, LLM services. We present Tarragon, a resilient MoE inference framework that confines the failures impact to individual workers while allowing the rest of the pipeline to continue making forward progress. Tarragon exploits the natural separation between the attention and expert computation in MoE-based transformers, treating attention workers (AWs) and expert workers (EWs) as distinct failure domains. Tarragon introduces a reconfigurable datapath to mask failures by rerouting requests to healthy workers. On top of this datapath, Tarragon implements a self-healing mechanism that relaxes the tightly synchronized execution of existing MoE frameworks. For stateful AWs, Tarragon performs asynchronous, incremental KV cache checkpointing with per-request restoration, and for stateless EWs, it leverages residual GPU memory to deploy shadow experts. These together keep recovery cost and recomputation overhead extremely low. Our evaluation shows that, compared to state-of-the-art MegaScale-Infer, Tarragon reduces failure-induced stalls by 160-213x (from ~64 s down to 0.3-0.4 s) while preserving performance when no failures occur.