mlsanityData to 14 Oct 2026

Placement Semantics for Distributed Deep Learning: A Systematic Framework for Analyzing Parallelism Strategies

arXiv 2601.02311 ↗First posted 5 Jan 2026cs.DC cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Training large language models requires distributing computation across many accelerators, yet practitioners select parallelism strategies (data, tensor, pipeline, ZeRO) through trial and error because no unified systematic framework predicts their behavior. We introduce placement semantics: each strategy is specified by how it places four training states (parameters, optimizer, gradients, activations) across devices using five modes (replicated, sharded, sharded-with-gather, materialized, offloaded). From placement alone, without implementation details, we derive memory consumption and communication volume. Our predictions match published results exactly: ZeRO-3 uses 8x less memory than data parallelism at 1.5x communication cost, as reported in the original paper. We prove two conditions (gradient integrity, state consistency) are necessary and sufficient for distributed training to match single-device results, and provide composition rules for combining strategies safely. The framework unifies ZeRO Stages 1-3, Fully Sharded Data Parallel (FSDP), tensor parallelism, and pipeline parallelism as instances with different placement choices.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 27 references that are in the map. First posted on arXiv 5 Jan 2026.

Builds on

19 of 27 references.

References · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  11. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  12. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
7 more
  1. 2017Mixed Precision Trainingcited by 1,156
  2. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  3. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  5. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  6. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  7. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 53

Built on by

Not cited yet.