mlsanityData to 7 Oct 2026

Scaling Distributed Machine Learning with In-Network Aggregation

arXiv 1903.06701 ↗First posted 22 Feb 2019cs.DC cs.LG cs.NIcited by 62

Not on the map. Why?

Abstract (arXiv)

Training machine learning models in parallel is an increasingly important workload. We accelerate distributed parallel training by designing a communication primitive that uses a programmable switch dataplane to execute a key step of the training process. Our approach, SwitchML, reduces the volume of exchanged data by aggregating the model updates from multiple workers in the network. We co-design the switch processing with the end-host protocols and ML frameworks to provide an efficient solution that speeds up training by up to 5.5× for a number of real-world benchmark models.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 11; 2023: 12; 2024: 13; 2025: 5; 2026: 7.

62 papers.

Sort:

By area:

  1. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  2. 2021EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learningcited by 53
  3. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  4. 2021Rethinking gradient sparsification as total error minimizationcited by 41
  5. 2023A Guide Through the Zoo of Biased SGDcited by 39
  6. 2021TACCL: Guiding Collective Algorithm Synthesis using Communication Sketchescited by 38
  7. 2023Momentum Provably Improves Error Feedback!cited by 31
  8. 2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
  9. 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
  10. 2022Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezingcited by 22
  11. 2021Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimizationcited by 21
  12. 2021Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Modelscited by 15
38 more
  1. 2023THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compressioncited by 15
  2. 2025Glia: A Human-Inspired AI for Automated Systems Design and Optimizationcited by 14
  3. 2022Federated Learning and Meta Learning: Approaches, Applications, and Directionscited by 12
  4. 2021Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategiescited by 9
  5. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 9
  6. 2021IntSGD: Adaptive Floatless Compression of Stochastic Gradientscited by 8
  7. 2022HammingMesh: A Network Topology for Large-Scale Deep Learningcited by 8
  8. 2024Swing: Short-cutting Rings for Higher Bandwidth Allreducecited by 8
  9. 2022QUIC-FL: Quick Unbiased Compression for Federated Learningcited by 7
  10. 2022Shifted Compression Framework: Generalizations and Improvementscited by 6
  11. 2022Impact of RoCE Congestion Control Policies on Distributed Training of DNNscited by 6
  12. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  13. 2024Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendationcited by 6
  14. 2024Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compressioncited by 6
  15. 2024Brain-on-Switch: Towards Advanced Intelligent Network Data Plane via NN-Driven Traffic Analysis at Line-Speedcited by 5
  16. 2022Distributed Methods with Absolute Compression and Error Compensationcited by 4
  17. 2023TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learningcited by 4
  18. 2024Accelerating Distributed Deep Learning using Lossless Homomorphic Compressioncited by 4
  19. 2025Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEscited by 4
  20. 2026DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reducecited by 4
  21. 2021LIBRA: Enabling Workload-aware Multi-dimensional Network Topology Optimization for Distributed Training of Large AI Modelscited by 3
  22. 2021Efficient Strong Scaling Through Burst Parallel Trainingcited by 3
  23. 2022IIsy: Practical In-Network Classificationcited by 3
  24. 2023Boosting Distributed Machine Learning Training Through Loss-tolerant Transmission Protocolcited by 3
  25. 2024Communication-efficient Vertical Federated Learning via Compressed Error Feedbackcited by 3
  26. 2021DeepReduce: A Sparse-tensor Communication Framework for Distributed Deep Learningcited by 2
  27. 2023OSP: Boosting Distributed Model Training with 2-stage Synchronizationcited by 2
  28. 2024Expediting In-Network Federated Learning by Voting-Based Consensus Model Compressioncited by 2
  29. 2024ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabricscited by 2
  30. 2025Accelerated Distributed Optimization with Compression and Error Feedbackcited by 2
  31. 2021MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Trainingcited by 1
  32. 2021Automatic Configuration for Optimal Communication Scheduling in DNN Trainingcited by 1
  33. 2022ByteComp: Revisiting Gradient Compression in Distributed Trainingcited by 1
  34. 2024Private Knowledge Sharing in Distributed Learning: A Surveycited by 1
  35. 2024LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Modelscited by 1
  36. 2024FRED: Flexible REduction-Distribution Interconnect and Communication Implementation for Wafer-Scale Distributed Training of DNN Modelscited by 1
  37. 2026Gradient Compression May Hurt Generalization: A Remedy by Synthetic Data Guided Sharpness Aware Minimizationcited by 1
  38. 2022Efficient Data-Plane Memory Scheduling for In-Network Aggregationcited by 0

50 most cited of 62 All 62 (JSON)