mlsanityData to 14 Oct 2026

Model Parallelism With Subnetwork Data Parallelism

arXiv 2507.09029 ↗First posted 11 Jul 2025cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Pre-training large neural networks at scale imposes heavy memory demands on accelerators and often requires costly communication. We introduce Subnetwork Data Parallelism (SDP), a distributed training framework that partitions a model into structured subnetworks trained across workers without exchanging activations. We study two complementary masking regimes: backward masking, which applies sparsity only in the backward step to retain unbiased gradients, and forward masking, which also removes parameters in the forward pass to deliver stronger efficiency gains while providing additional regularization. We further explore two subnetwork construction strategies: neuron level and block level, applied across both transformers and CNNs. In experiments spanning 1B LLaMA pre-training on FineWeb to ResNet-18 on CIFAR, SDP reduces per device memory usage by 28%-60% while maintaining or improving performance under FLOP-matched settings.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 11 Jul 2025.

Builds on

41 of 55 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2023Segment Anythingcited by 8,234
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  11. 2016Wide Residual Networkscited by 3,070
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
29 more
  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  4. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  6. 2020Adaptive Federated Optimizationcited by 1,046
  7. 2016Deep Networks with Stochastic Depthcited by 979
  8. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  9. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  10. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  11. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  12. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  13. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  14. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 163
  15. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  16. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  17. 2019Decoupled Greedy Learning of CNNscited by 78
  18. 2024Theia: Distilling Diverse Vision Foundation Models for Robot Learningcited by 61
  19. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 59
  20. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
  21. 2019Understanding Top-k Sparsification in Distributed Deep Learningcited by 56
  22. 2021Federated Dropout -- A Simple Approach for Enabling Federated Learning on Resource Constrained Devicescited by 56
  23. 2023PopulAtion Parameter Averaging (PAPA)cited by 23
  24. 2025Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertaintycited by 22
  25. 2021Enabling On-Device Training of Speech Recognition Models with Federated Dropoutcited by 12
  26. 2024WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Averagecited by 9
  27. 2022DADAO: Decoupled Accelerated Decentralized Asynchronous Optimizationcited by 8
  28. 2024PETRA: Parallel End-to-end Training with Reversible Architecturescited by 6
  29. 2025Accelerated Training through Iterative Gradient Propagation Along the Residual Pathcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Clustercited by 9
  2. 2026Heterogeneous Low-Bandwidth Pre-Training of LLMscited by 1
  3. 2026Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagationcited by 0