mlsanityData to 14 Oct 2026

Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Training

arXiv 2509.03018 ↗First posted 3 Sep 2025cs.DC cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Reliability is essential for ensuring efficiency in LLM training. However, many real-world reliability issues remain difficult to resolve, resulting in wasted resources and degraded model performance. Unfortunately, today's collective communication libraries operate as black boxes, hiding critical information needed for effective root cause analysis. We propose Mycroft, a lightweight distributed tracing and root cause analysis system designed to address previously hidden reliability issues in collective communication. Mycroft's key idea is to trace collective communication states and leverage internal control and data dependencies to resolve reliability problems in LLM training. Mycroft has been deployed at ByteDance for over six months to debug collective communication related issues at runtime. It detected anomalies within 15 seconds in 90% of cases and identified the root cause within 20 seconds in 60% of cases. We also conducted extensive fault injection experiments to demonstrate Mycroft's capability and efficiency.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 22 of its 61 references that are in the map. First posted on arXiv 3 Sep 2025.

Builds on

22 of 61 references.

References · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  4. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  5. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  6. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  7. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  8. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
  9. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  10. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  11. 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
  12. 2020Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systemscited by 38
10 more
  1. 2024Revisiting Reliability in Large-Scale Machine Learning Research Clusterscited by 28
  2. 2024ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Developmentcited by 24
  3. 2020Is Network the Bottleneck of Distributed Training?cited by 20
  4. 2024ReCycle: Resilient Training of Large DNNs using Pipeline Adaptationcited by 14
  5. 2023Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Tracescited by 13
  6. 2024Cloud Atlas: Efficient Fault Localization for Cloud Systems using Language Models and Causal Insightcited by 12
  7. 2024SuperBench: Improving Cloud AI Infrastructure Reliability with Proactive Validationcited by 11
  8. 2024Minder: Faulty Machine Detection for Large-scale Distributed Model Trainingcited by 11
  9. 2022Mystique: Enabling Accurate and Scalable Generation of Production AI Benchmarkscited by 5
  10. 2024Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelizationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 37
  2. 2025Role-Based Fault Tolerance System for LLM RL Post-Trainingcited by 2
  3. 2026LatencyPrism: Online Non-intrusive Latency Sculpting for SLO-Guaranteed LLM Inferencecited by 1
  4. 2025Detecting Anomalies in Machine Learning Infrastructure via Hardware Telemetrycited by 0
  5. 2026Enabling Performant and Flexible Model-Internal Observability for LLM Inferencecited by 0
  6. 2026SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Trainingcited by 0