mlsanityData to 7 Oct 2026

Tuning Collective Patterns to Alleviate Congestion in Shared AI Clusters

arXiv 2609.04417 ↗First posted 3 Sep 2026cs.NI cs.DC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Distributed AI training involves recurring rounds of data exchange between multiple pairs of GPU nodes. Slowdown in even one flow due to congestion can cause the entire communication round to slowdown. Current approaches for evading congestion in AI clusters assume global control over the entire workload (e.g. coordinating the schedule of all jobs) or assume infrastructural support (e.g. adaptive routing in switches). They are thus ill-suited in a shared cloud setting where AI jobs belonging to one user can face external congestion from other users' jobs or background traffic beyond its own control. In this paper, we build a system, REACT, that tunes the recurring pattern of data exchange between GPU nodes (known as communication collectives) in response to congestion. REACT works at the application (communication library) layer, where it detects congestion at runtime using readily available flow stats, and tunes the collective pattern to alleviate congestion - changing the set of incident flows while retaining the semantics of information exchange (e.g. selecting which node aggregates data in an AllReduce tree). REACT requires no explicit support from the underlying network infrastructure and can be unilaterally deployed by individual users in a shared cloud setting. We prototype REACT as a shim layer over NCCL, and evaluate it on a shared academic GPU cluster - enabling REACT improves communication performance (algorithm bandwidth) by 13%-38% under network congestion. Our simulations across a range of congestion scenarios further reveal up to 75% performance improvement, highlighting the effectiveness of our approach.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 59 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

16 of 59 references.

References · 16

  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  2. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  4. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
  5. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 68
  6. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 55
  7. 2019Themis: Fair and Efficient GPU Cluster Schedulingcited by 43
  8. 2021TACCL: Guiding Collective Algorithm Synthesis using Communication Sketchescited by 38
  9. 2020Synthesizing Optimal Collective Algorithmscited by 30
  10. 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
  11. 2023Rethinking Machine Learning Collective Communication as a Multi-Commodity Flow Problemcited by 11
  12. 2018Homa: A Receiver-Driven Low-Latency Transport Protocol Using Network Priorities (Complete Version)cited by 9
4 more
  1. 2023OptiReduce: Resilient and Tail-Optimal AllReduce for Distributed Deep Learning in the Cloudcited by 8
  2. 2024Swing: Short-cutting Rings for Higher Bandwidth Allreducecited by 8
  3. 2024REPS: Recycled Entropy Packet Spraying for Adaptive Load Balancing and Failure Mitigationcited by 6
  4. 2023Flock: Accurate network fault localization at scalecited by 2

Built on by

Not cited yet.