# Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters

arXiv: https://arxiv.org/abs/1706.03292 · HTML: https://mlsanity.com/paper/1706.03292/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2017.

## Built on by (most cited first, 30 of 57)

- [Accelerating Distributed MoE Training and Inference with Lina](https://mlsanity.com/paper/2210.17223/) (2022)
- [Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads](https://mlsanity.com/paper/2105.11118/) (2021)
- [Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs](https://mlsanity.com/paper/2204.12013/) (2022)
- [D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving](https://mlsanity.com/paper/2403.01876/) (2024)
- [Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezing](https://mlsanity.com/paper/2201.06227/) (2022)
- [AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learning](https://mlsanity.com/paper/2110.13005/) (2021)
- [HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environments](https://mlsanity.com/paper/2111.10635/) (2021)
- [Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression](https://mlsanity.com/paper/2301.09830/) (2023)
- [MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems](https://mlsanity.com/paper/2310.02784/) (2023)
- [BAGUA: Scaling up Distributed Learning with System Relaxations](https://mlsanity.com/paper/2107.01499/) (2021)
- [AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness](https://mlsanity.com/paper/2210.07297/) (2022)
- [dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Training](https://mlsanity.com/paper/2205.02473/) (2022)
- [Does compressing activations help model parallel training?](https://mlsanity.com/paper/2301.02654/) (2023)
- [A Survey and Empirical Evaluation of Parallel Deep Learning Frameworks](https://mlsanity.com/paper/2111.04949/) (2021)
- [Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioning](https://mlsanity.com/paper/2206.15143/) (2022)
- [Evaluation and Optimization of Gradient Compression for Distributed Deep Learning](https://mlsanity.com/paper/2306.08881/) (2023)
- [Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision](https://mlsanity.com/paper/2205.11913/) (2022)
- [Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks](https://mlsanity.com/paper/2107.06533/) (2021)
- [Efficient Direct-Connect Topologies for Collective Communications](https://mlsanity.com/paper/2202.03356/) (2022)
- [Enhancing Neural Training via a Correlated Dynamics Model](https://mlsanity.com/paper/2312.13247/) (2023)
- [OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusters](https://mlsanity.com/paper/2503.17469/) (2025)
- [Multi-model Machine Learning Inference Serving with GPU Spatial Partitioning](https://mlsanity.com/paper/2109.01611/) (2021)
- [FreshGNN: Reducing Memory Access via Stable Historical Embeddings for Graph Neural Network Training](https://mlsanity.com/paper/2301.07482/) (2023)
- [Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusters](https://mlsanity.com/paper/2205.09470/) (2022)
- [DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining](https://mlsanity.com/paper/2302.12445/) (2023)
- [Saturn: An Optimized Data System for Large Model Deep Learning Workloads](https://mlsanity.com/paper/2309.01226/) (2023)
- [Echo: Simulating Distributed Training At Scale](https://mlsanity.com/paper/2412.12487/) (2024)
- [FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training](https://mlsanity.com/paper/2510.00207/) (2025)
- [EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Training](https://mlsanity.com/paper/2511.10333/) (2025)
- [DynaComm: Accelerating Distributed CNN Training between Edges and Clouds through Dynamic Communication Scheduling](https://mlsanity.com/paper/2101.07968/) (2021)

## Abstract (arXiv)

Deep learning models can take weeks to train on a single GPU-equipped machine, necessitating scaling out DL training to a GPU-cluster. However, current distributed DL implementations can scale poorly due to substantial parameter synchronization over the network, because the high throughput of GPUs allows more data batches to be processed per unit time than CPUs, leading to more frequent network synchronization. We present Poseidon, an efficient communication architecture for distributed DL on GPUs. Poseidon exploits the layered model structures in DL programs to overlap communication and computation, reducing bursty network communication. Moreover, Poseidon uses a hybrid communication scheme that optimizes the number of bytes required to synchronize each layer, according to layer properties and the number of machines. We show that Poseidon is applicable to different DL frameworks by plugging Poseidon into Caffe and TensorFlow. We show that Poseidon enables Caffe and TensorFlow to achieve 15.5x speed-up on 16 single-GPU machines, even with limited bandwidth (10GbE) and the challenging VGG19-22K network for image classification. Moreover, Poseidon-enabled TensorFlow achieves 31.5x speed-up with 32 single-GPU machines on Inception-V3, a 50% improvement over the open-source TensorFlow (20x speed-up).