mlsanityData to 14 Oct 2026

SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training

arXiv 2608.11034 ↗First posted 11 Aug 2026cs.DC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In LLM pre-training, synchronization propagates rank-local stalls, slowdowns, and numerical errors into job-wide symptoms, obscuring their origin. Existing diagnosis often relies on in-process monitors that cannot report after the trainer blocks or terminates, or on post-mortem logs that preserve only synchronized symptoms; offline health tests lose the workload and operating conditions that triggered the failure. We present SCOUT, a unified runtime failure-localization framework built on one design principle: identify outliers through strict-majority consensus among equivalent replicas. SCOUT aligns replica progress, timing, and numerical evidence, then uses its Consensus Collective Communication (C3) abstraction to identify ranks whose compact signatures disagree with their peers. An out-of-band CPU observer remains responsive when training hangs, whereas in-situ replay exercises recurring stragglers and silent data corruption (SDC) beside the live job with its model state, kernels, allocations, communication path, and thermal and memory pressure present. Collective fingerprints expose rank-local protocol divergence. Clean replay coverage certifies checkpoint numerical integrity, preventing recovery from selecting state corrupted by SDC. SCOUT integrates with PyTorch, TorchTitan, Megatron-Core, and DeepSpeed without training-loop or framework-source modifications. SCOUT is open source at https://github.com/LMResiliency/lm-resiliency.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 53 references that are in the map. First posted on arXiv 11 Aug 2026.

Builds on

19 of 53 references.

References · 19

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  7. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  8. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  9. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  10. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 85
  11. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  12. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 31
7 more
  1. 2025Robust LLM Training Infrastructure at ByteDancecited by 16
  2. 2025Understanding Silent Data Corruption in LLM Trainingcited by 15
  3. 2025Understanding Stragglers in Large Model Training Using What-if Analysiscited by 12
  4. 2025Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checkscited by 10
  5. 2025Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Trainingcited by 6
  6. 2025TrainVerify: Equivalence-Based Verification for Distributed LLM Trainingcited by 3
  7. 2026Exploring Silent Data Corruption as a Reliability Challenge in LLM Trainingcited by 3

Built on by

Not cited yet.