# Tuning Collective Patterns to Alleviate Congestion in Shared AI Clusters

arXiv: https://arxiv.org/abs/2609.04417 · HTML: https://mlsanity.com/paper/2609.04417/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 59 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Builds on (most cited first, 16 of 16)

- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads](https://mlsanity.com/paper/1901.05758/) (2019)
- [Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning](https://mlsanity.com/paper/2008.12260/) (2020)
- [Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters](https://mlsanity.com/paper/1706.03292/) (2017)
- [Themis: Fair and Efficient GPU Cluster Scheduling](https://mlsanity.com/paper/1907.01484/) (2019)
- [TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches](https://mlsanity.com/paper/2111.04867/) (2021)
- [Synthesizing Optimal Collective Algorithms](https://mlsanity.com/paper/2008.08708/) (2020)
- [CASSINI: Network-Aware Job Scheduling in Machine Learning Clusters](https://mlsanity.com/paper/2308.00852/) (2023)
- [Rethinking Machine Learning Collective Communication as a Multi-Commodity Flow Problem](https://mlsanity.com/paper/2305.13479/) (2023)
- [Homa: A Receiver-Driven Low-Latency Transport Protocol Using Network Priorities (Complete Version)](https://mlsanity.com/paper/1803.09615/) (2018)
- [OptiReduce: Resilient and Tail-Optimal AllReduce for Distributed Deep Learning in the Cloud](https://mlsanity.com/paper/2310.06993/) (2023)
- [Swing: Short-cutting Rings for Higher Bandwidth Allreduce](https://mlsanity.com/paper/2401.09356/) (2024)
- [REPS: Recycled Entropy Packet Spraying for Adaptive Load Balancing and Failure Mitigation](https://mlsanity.com/paper/2407.21625/) (2024)
- [Flock: Accurate network fault localization at scale](https://mlsanity.com/paper/2305.03348/) (2023)

## Abstract (arXiv)

Distributed AI training involves recurring rounds of data exchange between multiple pairs of GPU nodes. Slowdown in even one flow due to congestion can cause the entire communication round to slowdown. Current approaches for evading congestion in AI clusters assume global control over the entire workload (e.g. coordinating the schedule of all jobs) or assume infrastructural support (e.g. adaptive routing in switches). They are thus ill-suited in a shared cloud setting where AI jobs belonging to one user can face external congestion from other users' jobs or background traffic beyond its own control. In this paper, we build a system, REACT, that tunes the recurring pattern of data exchange between GPU nodes (known as communication collectives) in response to congestion. REACT works at the application (communication library) layer, where it detects congestion at runtime using readily available flow stats, and tunes the collective pattern to alleviate congestion - changing the set of incident flows while retaining the semantics of information exchange (e.g. selecting which node aggregates data in an AllReduce tree). REACT requires no explicit support from the underlying network infrastructure and can be unilaterally deployed by individual users in a shared cloud setting. We prototype REACT as a shim layer over NCCL, and evaluate it on a shared academic GPU cluster - enabling REACT improves communication performance (algorithm bandwidth) by 13%-38% under network congestion. Our simulations across a range of congestion scenarios further reveal up to 75% performance improvement, highlighting the effectiveness of our approach.