mlsanityData to 14 Oct 2026

Horovod: fast and easy distributed deep learning in TensorFlow

arXiv 1802.05799 ↗First posted 15 Feb 2018cs.LG stat.MLcited by 279

Not on the map. Why?

Abstract (arXiv)

Training modern deep learning models requires large amounts of computation, often provided by GPUs. Scaling computation from one GPU to many can enable much faster training and research progress but entails two complications. First, the training library must support inter-GPU communication. Depending on the particular methods employed, this communication may entail anywhere from negligible to significant overhead. Second, the user must modify his or her training code to take advantage of inter-GPU communication. Depending on the training library's API, the modification required may be either significant or minimal. Existing methods for enabling multi-GPU training under the TensorFlow library entail non-negligible communication overhead and require users to heavily modify their model-building code, leading many researchers to avoid the whole mess and stick with slower single-GPU training. In this paper we introduce Horovod, an open source library that improves on both obstructions to scaling: it employs efficient inter-GPU communication via ring reduction and requires only a few lines of modification to user code, enabling faster, easier distributed training in TensorFlow. Horovod is available under the Apache 2.0 license at https://github.com/uber/horovod

Key facts. Cited by 279 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 80; 2022: 53; 2023: 52; 2024: 34; 2025: 28; 2026: 31.

279 papers.

Sort:

By area:

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  2. 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 608
  3. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  4. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 224
  5. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
  6. 2021A System for General In-Hand Object Re-Orientationcited by 165
  7. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  8. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  9. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  10. 2021SceneGen: Learning to Generate Realistic Traffic Scenescited by 84
  11. 2021Federated Quantum Machine Learningcited by 82
  12. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
38 more
  1. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  2. 2025Dion: Distributed Orthonormalized Updatescited by 68
  3. 2021Distributed Deep Learning in Open Collaborationscited by 58
  4. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  5. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  6. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 46
  7. 2021Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloadscited by 40
  8. 2021TACCL: Guiding Collective Algorithm Synthesis using Communication Sketchescited by 38
  9. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 35
  10. 2021An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systemscited by 32
  11. 2021Performance of a Geometric Deep Learning Pipeline for HL-LHC Particle Trackingcited by 30
  12. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 30
  13. 2024The Future of Large Language Model Pre-training is Federatedcited by 30
  14. 2021HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Frameworkcited by 28
  15. 2021An Efficient DP-SGD Mechanism for Large Scale NLP Modelscited by 27
  16. 2022FLUTE: A Scalable, Extensible Framework for High-Performance Federated Learning Simulationscited by 27
  17. 2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
  18. 2021Librispeech Transducer Model with Internal Language Model Prior Correctioncited by 26
  19. 2022Aspects of scaling and scalability for flow-based sampling of lattice QCDcited by 26
  20. 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
  21. 2024Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inferencecited by 25
  22. 2021MALib: A Parallel Framework for Population-based Multi-agent Reinforcement Learningcited by 23
  23. 2022Near-Optimal Sparse Allreduce for Distributed Deep Learningcited by 23
  24. 2021Czert -- Czech BERT-like Model for Language Representationcited by 22
  25. 2021Relative Molecule Self-Attention Transformercited by 21
  26. 2022Scalable algorithms for physics-informed neural and graph networkscited by 21
  27. 2021KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networkscited by 20
  28. 2022Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learningcited by 20
  29. 2023FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUscited by 20
  30. 2021Communication-efficient SGD: From Local SGD to One-Shot Averagingcited by 19
  31. 2022You May Not Need Ratio Clipping in PPOcited by 19
  32. 2022On Optimizing the Communication of Model Parallelismcited by 19
  33. 2021ResIST: Layer-Wise Decomposition of ResNets for Distributed Trainingcited by 18
  34. 2021TSM: Temporal Shift Module for Efficient and Scalable Video Understanding on Edge Devicecited by 17
  35. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
  36. 2022Data Debugging with Shapley Importance over End-to-End Machine Learning Pipelinescited by 17
  37. 2023FLINT: A Platform for Federated Learning Integrationcited by 17
  38. 2021Training Multilingual Pre-trained Language Model with Byte-level Subwordscited by 16

50 most cited of 279 All 279 (JSON)