mlsanityData to 14 Oct 2026

Verify Distributed Deep Learning Model Implementation Refinement with Iterative Relation Inference

arXiv 2508.09505 ↗First posted 13 Aug 2025cs.DC cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Distributed machine learning training and inference is common today because today's large models require more memory and compute than can be provided by a single GPU. Distributed models are generally produced by programmers who take a sequential model specification and apply several distribution strategies to distribute state and computation across GPUs. Unfortunately, bugs can be introduced in the process, and a distributed model implementation's outputs might differ from the sequential model's outputs. In this paper, we describe an approach to statically identify such bugs by checking model refinement, that is, can the sequential model's outputs be reconstructed from the distributed model's outputs? Our approach, implemented in GraphGuard, uses iterative rewriting to prove model refinement. Our approach can scale to today's large models and deployments: we evaluate it using GPT and Llama-3. Further, it provides actionable output that aids in bug localization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 44 references that are in the map. First posted on arXiv 13 Aug 2025.

Builds on

17 of 44 references.

References · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  9. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  10. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  11. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  12. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 112
5 more
  1. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  2. 2024Mirage: A Multi-Level Superoptimizer for Tensor Programscited by 41
  3. 2022NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilerscited by 30
  4. 2021Equality Saturation for Tensor Graph Superoptimizationcited by 27
  5. 2020Graph-Based Fuzz Testing for Deep Learning Inference Enginecited by 7

Built on by

Not cited yet.