# FireCaffe: near-linear acceleration of deep neural network training on compute clusters

arXiv: https://arxiv.org/abs/1511.00175 · HTML: https://mlsanity.com/paper/1511.00175/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2015.

## Built on by (most cited first, 25 of 25)

- [See through Gradients: Image Batch Recovery via GradInversion](https://mlsanity.com/paper/2104.07586/) (2021)
- [FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction](https://mlsanity.com/paper/2203.11751/) (2022)
- [Towards Efficient and Scalable Sharpness-Aware Minimization](https://mlsanity.com/paper/2203.02714/) (2022)
- [CAFE: Catastrophic Data Leakage in Vertical Federated Learning](https://mlsanity.com/paper/2110.15122/) (2021)
- [GRNN: Generative Regression Neural Network -- A Data Leakage Attack for Federated Learning](https://mlsanity.com/paper/2105.00529/) (2021)
- [On the Utility of Gradient Compression in Distributed Training Systems](https://mlsanity.com/paper/2103.00543/) (2021)
- [TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernels](https://mlsanity.com/paper/2207.06343/) (2022)
- [FedPAGE: A Fast Local Stochastic Gradient Method for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2108.04755/) (2021)
- [DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devices](https://mlsanity.com/paper/2309.05015/) (2023)
- [Concurrent Adversarial Learning for Large-Batch Training](https://mlsanity.com/paper/2106.00221/) (2021)
- [FA-GAN: Fused Attentive Generative Adversarial Networks for MRI Image Super-Resolution](https://mlsanity.com/paper/2108.03920/) (2021)
- [Manas: Mining Software Repositories to Assist AutoML](https://mlsanity.com/paper/2112.03395/) (2021)
- [Partitioning sparse deep neural networks for scalable training and inference](https://mlsanity.com/paper/2104.11805/) (2021)
- [Gradient Inversion Attack on Graph Neural Networks](https://mlsanity.com/paper/2411.19440/) (2024)
- [Analysing the Influence of Attack Configurations on the Reconstruction of Medical Images in Federated Learning](https://mlsanity.com/paper/2204.13808/) (2022)
- [Layer-Parallel Training of Residual Networks with Auxiliary-Variable Networks](https://mlsanity.com/paper/2112.05387/) (2021)
- [On the Permanence of Backdoors in Evolving Models](https://mlsanity.com/paper/2206.04677/) (2022)
- [Is Diffusion Model Safe? Severe Data Leakage via Gradient-Guided Diffusion Model](https://mlsanity.com/paper/2406.09484/) (2024)
- [Temporal Knowledge Distillation for Time-Sensitive Financial Services Applications](https://mlsanity.com/paper/2312.16799/) (2023)
- [Scaling Up Deep Clustering Methods Beyond ImageNet-1K](https://mlsanity.com/paper/2406.01203/) (2024)
- [Seamless Integration: Sampling Strategies in Federated Learning Systems](https://mlsanity.com/paper/2408.09545/) (2024)
- [Cloud Collectives: Towards Cloud-aware Collectives forML Workloads with Rank Reordering](https://mlsanity.com/paper/2105.14088/) (2021)
- [Privacy protection based on mask template](https://mlsanity.com/paper/2202.06250/) (2022)
- [Routing for Large ML Models](https://mlsanity.com/paper/2503.05324/) (2025)
- [SpecGradFilter: A Spectral Gradient Filtering Framework for Taming Federated Heterogeneity](https://mlsanity.com/paper/2607.04189/) (2026)

## Abstract (arXiv)

Long training times for high-accuracy deep neural networks (DNNs) impede research into new DNN architectures and slow the development of high-accuracy DNNs. In this paper we present FireCaffe, which successfully scales deep neural network training across a cluster of GPUs. We also present a number of best practices to aid in comparing advancements in methods for scaling and accelerating the training of deep neural networks. The speed and scalability of distributed algorithms is almost always limited by the overhead of communicating between servers; DNN training is not an exception to this rule. Therefore, the key consideration here is to reduce communication overhead wherever possible, while not degrading the accuracy of the DNN models that we train. Our approach has three key pillars. First, we select network hardware that achieves high bandwidth between GPU servers -- Infiniband or Cray interconnects are ideal for this. Second, we consider a number of communication algorithms, and we find that reduction trees are more efficient and scalable than the traditional parameter server approach. Third, we optionally increase the batch size to reduce the total quantity of communication during DNN training, and we identify hyperparameters that allow us to reproduce the small-batch accuracy while training with large batch sizes. When training GoogLeNet and Network-in-Network on ImageNet, we achieve a 47x and 39x speedup, respectively, when training on a cluster of 128 GPUs.