mlsanityData to 14 Oct 2026

Reliable and Resilient Collective Communication Library for LLM Training and Serving

arXiv 2512.25059 ↗First posted 31 Dec 2025cs.DC cs.LG cs.NIcited by 4

Not on the map. Why?

Abstract (arXiv)

Modern ML training and inference now span tens to tens of thousands of GPUs, where network faults can waste 10--15% of GPU hours due to slow recovery. Common network errors and link fluctuations trigger timeouts that often terminate entire jobs, forcing expensive checkpoint rollback during training and request reprocessing during inference. We present R2CCL, a fault-tolerant communication library that provides lossless, low-overhead failover by exploiting multi-NIC hardware. R2CCL performs rapid connection migration, bandwidth-aware load redistribution, and resilient collective algorithms to maintain progress under failures. We evaluate R2CCL on two 8-GPU H100 InfiniBand servers and via large-scale ML simulators modeling hundreds of GPUs with diverse failure patterns. Experiments show that R2CCL is highly robust to NIC failures, incurring less than 1% training and less than 3% inference overheads. R2CCL outperforms baselines AdapCC and DejaVu by 12.18× and 47×, respectively.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 16 of its 32 references that are in the map. First posted on arXiv 31 Dec 2025.

Builds on

16 of 32 references.

References · 16

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  4. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  5. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 108
  6. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 52
  7. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  8. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  9. 2021TACCL: Guiding Collective Algorithm Synthesis using Communication Sketchescited by 38
  10. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 31
  11. 2024Revisiting Reliability in Large-Scale Machine Learning Research Clusterscited by 28
  12. 2023Unicron: Economizing Self-Healing LLM Training at Scalecited by 16
4 more
  1. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 15
  2. 2024ReCycle: Resilient Training of Large DNNs using Pipeline Adaptationcited by 14
  3. 2023Rethinking Machine Learning Collective Communication as a Multi-Commodity Flow Problemcited by 11
  4. 2024Real-time Vehicle-to-Vehicle Communication Based Network Cooperative Control System through Distributed Database and Multimodal Perception: Demonstrated in Crossroadscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Towards Resiliency in Large Language Model Serving with KevlarFlowcited by 3
  2. 2026Enabling Performant and Flexible Model-Internal Observability for LLM Inferencecited by 0
  3. 2026ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workloadcited by 0
  4. 2026Don't Let a Few Network Failures Slow the Entire AllReducecited by 0