mlsanityData to 14 Oct 2026

Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters

arXiv 1706.03292 ↗First posted 11 Jun 2017cs.LG cs.CV cs.DCcited by 57

Not on the map. Why?

Abstract (arXiv)

Deep learning models can take weeks to train on a single GPU-equipped machine, necessitating scaling out DL training to a GPU-cluster. However, current distributed DL implementations can scale poorly due to substantial parameter synchronization over the network, because the high throughput of GPUs allows more data batches to be processed per unit time than CPUs, leading to more frequent network synchronization. We present Poseidon, an efficient communication architecture for distributed DL on GPUs. Poseidon exploits the layered model structures in DL programs to overlap communication and computation, reducing bursty network communication. Moreover, Poseidon uses a hybrid communication scheme that optimizes the number of bytes required to synchronize each layer, according to layer properties and the number of machines. We show that Poseidon is applicable to different DL frameworks by plugging Poseidon into Caffe and TensorFlow. We show that Poseidon enables Caffe and TensorFlow to achieve 15.5x speed-up on 16 single-GPU machines, even with limited bandwidth (10GbE) and the challenging VGG19-22K network for image classification. Moreover, Poseidon-enabled TensorFlow achieves 31.5x speed-up with 32 single-GPU machines on Inception-V3, a 50% improvement over the open-source TensorFlow (20x speed-up).

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 19; 2022: 15; 2023: 11; 2024: 2; 2025: 5; 2026: 5.

57 papers.

Sort:

By area:

  1. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  2. 2021Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threadscited by 51
  3. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  4. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  5. 2022Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezingcited by 22
  6. 2021AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learningcited by 15
  7. 2021HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environmentscited by 15
  8. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 15
  9. 2023MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systemscited by 14
  10. 2021BAGUA: Scaling up Distributed Learning with System Relaxationscited by 13
  11. 2022AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awarenesscited by 13
  12. 2022dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Trainingcited by 12
38 more
  1. 2023Does compressing activations help model parallel training?cited by 12
  2. 2021A Survey and Empirical Evaluation of Parallel Deep Learning Frameworkscited by 7
  3. 2022Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioningcited by 7
  4. 2023Evaluation and Optimization of Gradient Compression for Distributed Deep Learningcited by 7
  5. 2022Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Visioncited by 6
  6. 2021Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Taskscited by 5
  7. 2022Efficient Direct-Connect Topologies for Collective Communicationscited by 5
  8. 2023Enhancing Neural Training via a Correlated Dynamics Modelcited by 5
  9. 2025OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusterscited by 5
  10. 2021Multi-model Machine Learning Inference Serving with GPU Spatial Partitioningcited by 4
  11. 2023FreshGNN: Reducing Memory Access via Stable Historical Embeddings for Graph Neural Network Trainingcited by 4
  12. 2022Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusterscited by 3
  13. 2023DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipeliningcited by 3
  14. 2023Saturn: An Optimized Data System for Large Model Deep Learning Workloadscited by 2
  15. 2024Echo: Simulating Distributed Training At Scalecited by 2
  16. 2025FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Trainingcited by 2
  17. 2025EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Trainingcited by 2
  18. 2021DynaComm: Accelerating Distributed CNN Training between Edges and Clouds through Dynamic Communication Schedulingcited by 1
  19. 2021Peering Beyond the Gradient Veil with Distributed Auto Differentiationcited by 1
  20. 2021MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Trainingcited by 1
  21. 2021EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networkscited by 1
  22. 2021Automatic Configuration for Optimal Communication Scheduling in DNN Trainingcited by 1
  23. 2022ByteComp: Revisiting Gradient Compression in Distributed Trainingcited by 1
  24. 2021CrossoverScheduler: Overlapping Multiple Distributed Training Applications in a Crossover Mannercited by 0
  25. 2021Towards Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Frameworkcited by 0
  26. 2021CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensationcited by 0
  27. 2021SINGA-Easy: An Easy-to-Use Framework for MultiModal Analysiscited by 0
  28. 2021VisualTextRank: Unsupervised Graph-based Content Extraction for Automating Ad Text to Image Searchcited by 0
  29. 2021Text is NOT Enough: Integrating Visual Impressions into Open-domain Dialogue Generationcited by 0
  30. 2021Scheduling Optimization Techniques for Neural Network Trainingcited by 0
  31. 2022Efficient Data-Plane Memory Scheduling for In-Network Aggregationcited by 0
  32. 2022Discovering Personalized Semantics for Soft Attributes in Recommender Systems using Concept Activation Vectorscited by 0
  33. 2022Reinforced MOOCs Concept Recommendation in Heterogeneous Information Networkscited by 0
  34. 2022Explanation-Guided Fairness Testing through Genetic Algorithmcited by 0
  35. 2022Using Graph Neural Networks for Program Terminationcited by 0
  36. 2023Expediting Distributed DNN Training with Device Topology-Aware Graph Deploymentcited by 0
  37. 2023Cloudless-Training: A Framework to Improve Efficiency of Geo-Distributed ML Trainingcited by 0
  38. 2023Tribe or Not? Critical Inspection of Group Differences Using TribalGramcited by 0

50 most cited of 57 All 57 (JSON)