# Distributed TensorFlow with MPI

arXiv: https://arxiv.org/abs/1603.02339 · HTML: https://mlsanity.com/paper/1603.02339/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Mar 2016.

## Built on by (most cited first, 4 of 4)

- [From Distributed Machine Learning to Federated Learning: A Survey](https://mlsanity.com/paper/2104.14362/) (2021)
- [Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines](https://mlsanity.com/paper/2202.08679/) (2022)
- [Cloud Collectives: Towards Cloud-aware Collectives forML Workloads with Rank Reordering](https://mlsanity.com/paper/2105.14088/) (2021)
- [Toward efficient resource utilization at edge nodes in federated learning](https://mlsanity.com/paper/2309.10367/) (2023)

## Abstract (arXiv)

Machine Learning and Data Mining (MLDM) algorithms are becoming increasingly important in analyzing large volume of data generated by simulations, experiments and mobile devices. With increasing data volume, distributed memory systems (such as tightly connected supercomputers or cloud computing systems) are becoming important in designing in-memory and massively parallel MLDM algorithms. Yet, the majority of open source MLDM software is limited to sequential execution with a few supporting multi-core/many-core execution. In this paper, we extend recently proposed Google TensorFlow for execution on large scale clusters using Message Passing Interface (MPI). Our approach requires minimal changes to the TensorFlow runtime -- making the proposed implementation generic and readily usable to increasingly large users of TensorFlow. We evaluate our implementation using an InfiniBand cluster and several well knowndatasets. Our evaluation indicates the efficiency of our proposed implementation.