mlsanityData to 7 Oct 2026

Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checks

arXiv 2506.14813 ↗First posted 6 Jun 2025cs.LG cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Training deep learning (DL) models is a complex process, making it prone to silent errors that are challenging to detect and diagnose. This paper presents TRAINCHECK, a framework that takes a proactive checking approach to address silent training errors. TRAINCHECK automatically infers invariants tailored for DL training. It uses these invariants to proactively detect silent errors during the training process while providing debugging help. To evaluate TRAINCHECK, we reproduce 20 real-world silent training errors with diverse root causes. TRAINCHECK successfully detects 18 errors within a single training iteration. It also uncovers 6 unknown bugs in popular training libraries that lead to silent errors.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 13 of its 46 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

13 of 46 references.

References · 13

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  3. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 307
  4. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 249
  5. 2020Hyper-Parameter Optimization: A Review of Algorithms and Applicationscited by 114
  6. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  7. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
  8. 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
  9. 2022NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilerscited by 29
  10. 2022Toward Understanding Deep Learning Framework Bugscited by 26
  11. 2023NeuRI: Diversifying DNN Generation via Inductive Rule Inferencecited by 6
  12. 2023Reliability Assurance for Deep Neural Network Architectures Against Numerical Defectscited by 3
1 more
  1. 2021A Static Analyzer for Detecting Tensor Shape Errors in Deep Neural Network Training Codecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025TrainVerify: Equivalence-Based Verification for Distributed LLM Trainingcited by 3
  2. 2025Role-Based Fault Tolerance System for LLM RL Post-Trainingcited by 2
  3. 2025TTrace: Lightweight Error Checking and Diagnosis for Distributed Trainingcited by 1
  4. 2025Verifying Computational Graphs in Production-Grade Distributed Machine Learning Frameworkscited by 0
  5. 2026DepRadar: Agentic Coordination for Context Aware Defect Impact Analysis in Deep Learning Librariescited by 0
  6. 2026Silent Inconsistency in Data-Parallel Full Fine-Tuning: Diagnosing Worker-Level Optimization Misalignmentcited by 0
  7. 2026Demystifying the Silence of Correctness Bugs in PyTorch Compilercited by 0
  8. 2026Ekka: Automated Diagnosis of Silent Errors in LLM Inferencecited by 0
  9. 2026SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Trainingcited by 0
  10. 2026Trains but Doesn't Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineerscited by 0