# Node-Aware Improvements to Allreduce

arXiv: https://arxiv.org/abs/1910.09650 · HTML: https://mlsanity.com/paper/1910.09650/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2019.

## Built on by (most cited first, 1 of 1)

- [TerraceMoE: A Cost Model for Hierarchical MoE All-to-All Communication](https://mlsanity.com/paper/2608.27874/) (2026)

## Abstract (arXiv)

The textttMPIAllreduce collective operation is a core kernel of many parallel codebases, particularly for reductions over a single value per process. The commonly used allreduce recursive-doubling algorithm obtains the lower bound message count, yielding optimality for small reduction sizes based on node-agnostic performance models. However, this algorithm yields duplicate messages between sets of nodes. Node-aware optimizations in MPICH remove duplicate messages through use of a single master process per node, yielding a large number of inactive processes at each inter-node step. In this paper, we present an algorithm that uses the multiple processes available per node to reduce the maximum number of inter-node messages communicated by a single process, improving the performance of allreduce operations, particularly for small message sizes.