mlsanityData to 14 Oct 2026

TrainVerify: Equivalence-Based Verification for Distributed LLM Training

arXiv 2506.15961 ↗First posted 19 Jun 2025cs.DC cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Training large language models (LLMs) at scale requires parallel execution across thousands of devices, incurring enormous computational costs. Yet, these costly distributed trainings are rarely verified, leaving them prone to silent errors and potentially wasting millions of GPU hours. We introduce TrainVerify, a system for verifiable distributed training of LLMs. Given a deep learning model's logical specification as the ground truth, TrainVerify formally verifies that a distributed parallel execution plan is mathematically equivalent to it. Direct verification is notoriously difficult due to the sheer scale of LLMs which often involves billions of variables and highly intricate computation graphs. Therefore, TrainVerify introduces shape-reduction techniques and a stage-wise parallel verification algorithm that significantly reduces complexity while preserving formal correctness. TrainVerify scales to frontier LLMs, including the successful verification of the Llama3 (405B) and DeepSeek-V3 (671B) training plans.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 23 of its 81 references that are in the map. First posted on arXiv 19 Jun 2025.

Builds on

23 of 81 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  9. 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 744
  10. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  11. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  12. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
11 more
  1. 2019Algorithms for Verifying Deep Neural Networkscited by 221
  2. 2020Ansor: Generating High-Performance Tensor Programs for Deep Learningcited by 201
  3. 2024The rising costs of training frontier AI modelscited by 95
  4. 2015DeepID3: Face Recognition with Very Deep Neural Networkscited by 62
  5. 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
  6. 2022NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilerscited by 30
  7. 2021Maximizing Parallelism in Distributed Training for Huge Neural Networkscited by 23
  8. 2021OneFlow: Redesign the Distributed Deep Learning Framework from Scratchcited by 15
  9. 2025Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checkscited by 10
  10. 2024Revisiting Differential Verification: Equivalence Verification with Confidencecited by 4
  11. 2024Revealing Floating-Point Accumulation Orders in Software/Hardware Implementationscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Role-Based Fault Tolerance System for LLM RL Post-Trainingcited by 2
  2. 2025Verifying Computational Graphs in Production-Grade Distributed Machine Learning Frameworkscited by 0
  3. 2026SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Trainingcited by 0