# Minder: Faulty Machine Detection for Large-scale Distributed Model Training

arXiv: https://arxiv.org/abs/2411.01791 · HTML: https://mlsanity.com/paper/2411.01791/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 79 references that are in the map.
- First posted on arXiv 4 Nov 2024.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications](https://mlsanity.com/paper/1802.03903/) (2018)
- [Characterization of Large Language Model Development in the Datacenter](https://mlsanity.com/paper/2403.07648/) (2024)
- [A Nonparametric Approach for Multiple Change Point Analysis of Multivariate Data](https://mlsanity.com/paper/1306.4933/) (2013)
- [Unicron: Economizing Self-Healing LLM Training at Scale](https://mlsanity.com/paper/2401.00134/) (2023)
- [SuperBench: Improving Cloud AI Infrastructure Reliability with Proactive Validation](https://mlsanity.com/paper/2402.06194/) (2024)
- [Collie: Finding Performance Anomalies in RDMA Subsystems](https://mlsanity.com/paper/2304.11467/) (2023)

## Built on by (most cited first, 11 of 11)

- [Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Training](https://mlsanity.com/paper/2509.03018/) (2025)
- [Training LLMs with Fault Tolerant HSDP on 100,000 GPUs](https://mlsanity.com/paper/2602.00277/) (2026)
- [Role-Based Fault Tolerance System for LLM RL Post-Training](https://mlsanity.com/paper/2512.22492/) (2025)
- [A Feature Engineering Approach for Business Impact-Oriented Failure Detection in Distributed Instant Payment Systems](https://mlsanity.com/paper/2510.21710/) (2025)
- [LatencyPrism: Online Non-intrusive Latency Sculpting for SLO-Guaranteed LLM Inference](https://mlsanity.com/paper/2601.09258/) (2026)
- [PropLLM: Propagation-Aware Scene Reconstruction for Network Fault Diagnosis](https://mlsanity.com/paper/2606.00582/) (2026)
- [Alchemist: Towards the Design of Efficient Online Continual Learning System](https://mlsanity.com/paper/2503.01066/) (2025)
- [CCL-D: A High-Precision Diagnostic System for Slow and Hang Anomalies in Large-Scale Model Training](https://mlsanity.com/paper/2605.04478/) (2026)
- [From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs](https://mlsanity.com/paper/2605.09370/) (2026)
- [Enabling Performant and Flexible Model-Internal Observability for LLM Inference](https://mlsanity.com/paper/2605.11093/) (2026)
- [Leto: Fast In-Place Recovery for LLM Training on Surviving Hardware](https://mlsanity.com/paper/2610.00687/) (2026)

## Abstract (arXiv)

Large-scale distributed model training requires simultaneous training on up to thousands of machines. Faulty machine detection is critical when an unexpected fault occurs in a machine. From our experience, a training task can encounter two faults per day on average, possibly leading to a halt for hours. To address the drawbacks of the time-consuming and labor-intensive manual scrutiny, we propose Minder, an automatic faulty machine detector for distributed training tasks. The key idea of Minder is to automatically and efficiently detect faulty distinctive monitoring metric patterns, which could last for a period before the entire training task comes to a halt. Minder has been deployed in our production environment for over one year, monitoring daily distributed training tasks where each involves up to thousands of machines. In our real-world fault detection scenarios, Minder can accurately and efficiently react to faults within 3.6 seconds on average, with a precision of 0.904 and F1-score of 0.893.