mlsanityData to 14 Oct 2026

Piper: A Programmable Distributed Training System

arXiv 2606.11169 ↗First posted 9 Jun 2026cs.DC cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large-scale model training increasingly relies on composing multiple parallelism strategies, such as data, pipeline, and expert parallelism, together with memory-saving optimizations like ZeRO. Deployed systems for foundation model pretraining often rely on human experts to manually design a high-level parallelism strategy then implement the corresponding low-level execution strategy, making it difficult to adapt the system to new strategies. Meanwhile, many general-purpose frameworks are more flexible but their implementations are still tied to a fixed set of common parallelism strategies, making it challenging to integrate state-of-the-art strategies. We present Piper, a user-controllable distributed training system that decouples the strategy from the runtime implementation. Piper allows users to declare a comprehensive distributed training strategy with a small set of model annotations and scheduling directives. Each directive applies a transformation on Piper's intermediate representation (IR), a unified global training DAG that represents all computation and communication. Using this IR, Piper compiles per-device execution plans and executes them with a distributed runtime agnostic to the strategy. We show that the combined system maintains performance parity on commonly available strategies such as ZeRO, while also enabling additional performance and memory efficiency gains through joint scheduling of compute and communication in composed parallelism strategies such as DeepSeek-V3's DualPipe.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 51 references that are in the map. First posted on arXiv 9 Jun 2026.

Builds on

17 of 51 references.

References · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  5. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  6. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  7. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 111
  8. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 85
  9. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 82
  10. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 57
  11. 2018Supporting Very Large Models using Automatic Dataflow Graph Partitioningcited by 43
  12. 2022Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Modelscited by 11
5 more
  1. 2023Slapo: A Schedule Language for Progressive Optimization of Large Deep Learning Model Trainingcited by 11
  2. 2025Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimizationcited by 7
  3. 2025DeepCompile: A Compiler-Driven Approach to Optimizing Distributed Deep Learning Trainingcited by 2
  4. 2024Scaling Deep Learning Training with MPMD Pipeline Parallelismcited by 1
  5. 2026AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelismcited by 1

Built on by

Not cited yet.