mlsanityData to 7 Oct 2026

HammingMesh: A Network Topology for Large-Scale Deep Learning

arXiv 2209.01346 ↗First posted 3 Sep 2022cs.DC cs.AI cs.ARcited by 8

Not on the map. Why?

Abstract (arXiv)

Numerous microarchitectural optimizations unlocked tremendous processing power for deep neural networks that in turn fueled the AI revolution. With the exhaustion of such optimizations, the growth of modern AI is now gated by the performance of training systems, especially their data movement. Instead of focusing on single accelerators, we investigate data-movement characteristics of large-scale training at full system scale. Based on our workload analysis, we design HammingMesh, a novel network topology that provides high bandwidth at low cost with high job scheduling flexibility. Specifically, HammingMesh can support full bandwidth and isolation to deep learning training jobs with two dimensions of parallelism. Furthermore, it also supports high global bandwidth for generic traffic. Thus, HammingMesh will power future large-scale deep learning systems with extreme bandwidth requirements.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 38 of its 87 references that are in the map. First posted on arXiv 3 Sep 2022.

Builds on

38 of 87 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  7. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  8. 2017Large Batch Training of Convolutional Networkscited by 529
  9. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  10. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 427
  11. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  12. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
26 more
  1. 2018The Convergence of Sparsified Gradient Methodscited by 228
  2. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 199
  3. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  4. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 154
  5. 2017Revisiting Distributed Synchronous SGDcited by 142
  6. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
  7. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  8. 2020Local SGD: Unified Theory and New Efficient Methodscited by 88
  9. 2019The Architectural Implications of Facebook's DNN-based Personalized Recommendationcited by 83
  10. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  11. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  12. 2019Scaling Distributed Machine Learning with In-Network Aggregationcited by 62
  13. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 54
  14. 2018SparCML: High-Performance Sparse Communication for Machine Learningcited by 34
  15. 2020Survey of Machine Learning Acceleratorscited by 29
  16. 2020Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platformscited by 25
  17. 2022Near-Optimal Sparse Allreduce for Distributed Deep Learningcited by 23
  18. 2019Red-blue pebbling revisited: near optimal parallel matrix-matrix multiplicationcited by 21
  19. 2018CosmoFlow: Using Deep Learning to Learn the Universe at Scalecited by 14
  20. 2020An In-Depth Analysis of the Slingshot Interconnectcited by 13
  21. 2020Breaking (Global) Barriers in Parallel Stochastic Optimization with Wait-Avoiding Group Averagingcited by 9
  22. 2019Improving Strong-Scaling of CNN Training by Exploiting Finer-Grained Parallelismcited by 8
  23. 2021Flare: Flexible In-Network Allreducecited by 8
  24. 2019Slim Fly: A Cost Effective Low-Diameter Network Topologycited by 7
  25. 20170.5 Petabyte Simulation of a 45-Qubit Quantum Circuitcited by 2
  26. 2021On the Parallel I/O Optimality of Linear Algebra Kernels: Near-Optimal Matrix Factorizationscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 3.

8 papers.

  1. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 9
  2. 2024Swing: Short-cutting Rings for Higher Bandwidth Allreducecited by 8
  3. 2026DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reducecited by 4
  4. 2025Mixtera: A Data Plane for Foundation Model Trainingcited by 2
  5. 2025Cluster Topology-Driven Placement of Experts Reduces Network Traffic in MoE Inferencecited by 1
  6. 2025Bine Trees: Enhancing Collective Operations by Optimizing Communication Localitycited by 1
  7. 2026Maia 200: A Software Defined Dataflow System for Large-scale AI Accelerationcited by 0
  8. 2026Accelerating Sharded Data Parallelism at Scale with Federated Learningcited by 0