# HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis

arXiv: https://arxiv.org/abs/2401.05965 · HTML: https://mlsanity.com/paper/2401.05965/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jan 2024.

## Built on by (most cited first, 8 of 8)

- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow](https://mlsanity.com/paper/2406.01566/) (2024)
- [Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling](https://mlsanity.com/paper/2410.17043/) (2024)
- [HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUs](https://mlsanity.com/paper/2504.03871/) (2025)
- [gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling](https://mlsanity.com/paper/2504.14775/) (2025)
- [Spatio-Temporal Parallelism for Diffusion Model Inference on Heterogeneous Multi-GPU Systems](https://mlsanity.com/paper/2509.04719/) (2025)
- [Nonuniform-Tensor-Parallelism: Mitigating GPU failure impact for Scaled-up LLM Training](https://mlsanity.com/paper/2504.06095/) (2025)
- [Global Simulation-Guided Dynamic Operator Scheduling for Efficient Multi-Tenant Model Serving](https://mlsanity.com/paper/2608.15762/) (2026)

## Abstract (arXiv)

Single-Program-Multiple-Data (SPMD) parallelism has recently been adopted to train large deep neural networks (DNNs). Few studies have explored its applicability on heterogeneous clusters, to fully exploit available resources for large model learning. This paper presents OurSystem, an automated system designed to expedite SPMD DNN training on heterogeneous clusters. OurSystem jointly optimizes the tensor sharding strategy, sharding ratios across heterogeneous devices and the communication methods for tensor exchanges for optimized distributed training with SPMD parallelism. We novelly formulate model partitioning as a program synthesis problem, in which we generate a distributed program from scratch on a distributed instruction set that semantically resembles the program designed for a single device, and systematically explore the solution space with an A*-based search algorithm. We derive the optimal tensor sharding ratios by formulating it as a linear programming problem. Additionally, OurSystem explores tensor communication optimization in a heterogeneous cluster and integrates it as part of the program synthesis process, for automatically choosing optimal collective communication primitives and applying sufficient factor broadcasting technique. Extensive experiments on representative workloads demonstrate that OurSystem achieves up to 2.41x speed-up on heterogeneous clusters.