mlsanityData to 7 Oct 2026

MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems

arXiv 2310.02784 ↗First posted 4 Oct 2023cs.DC cs.AR cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Training and deploying large-scale machine learning models is time-consuming, requires significant distributed computing infrastructures, and incurs high operational costs. Our analysis, grounded in real-world large model training on datacenter-scale infrastructures, reveals that 14~32% of all GPU hours are spent on communication with no overlapping computation. To minimize this outstanding communication latency and other inherent at-scale inefficiencies, we introduce an agile performance modeling framework, MAD-Max. This framework is designed to optimize parallelization strategies and facilitate hardware-software co-design opportunities. Through the application of MAD-Max to a suite of real-world large-scale ML models on state-of-the-art GPU clusters, we showcase potential throughput enhancements of up to 2.24x for pre-training and up to 5.2x for inference scenarios, respectively.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 53 of its 84 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

53 of 84 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  11. 2017Neural Collaborative Filteringcited by 1,108
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
38 more
  1. 2016Wide & Deep Learning for Recommender Systemscited by 745
  2. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  3. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  4. 2017Deep Interest Network for Click-Through Rate Predictioncited by 541
  5. 2022Efficiently Scaling Transformer Inferencecited by 529
  6. 2017Deep & Cross Network for Ad Click Predictionscited by 392
  7. 2018Deep Interest Evolution Network for Click-Through Rate Predictioncited by 336
  8. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  9. 2020DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systemscited by 326
  10. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  11. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  12. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 183
  13. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  14. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  15. 2019Behavior Sequence Transformer for E-commerce Recommendation in Alibabacited by 128
  16. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  17. 2023ChatGPT is not all you need. A State of the Art Review of large Generative AI modelscited by 85
  18. 2019The Architectural Implications of Facebook's DNN-based Personalized Recommendationcited by 83
  19. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
  20. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  21. 2020Chasing Carbon: The Elusive Environmental Footprint of Computingcited by 71
  22. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 55
  23. 2020DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inferencecited by 54
  24. 2019RecNMP: Accelerating Personalized Recommendation with Near-Memory Processingcited by 49
  25. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  26. 2020Understanding Training Efficiency of Deep Learning Recommendation Models at Scalecited by 43
  27. 2019TensorDIMM: A Practical Near-Memory Processing Architecture for Embeddings and Tensor Operations in Deep Learningcited by 42
  28. 2021RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inferencecited by 35
  29. 2022On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Modelscited by 34
  30. 2022TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobscited by 30
  31. 2022RecShard: Statistical Feature-Based Memory Optimization for Industry-Scale Neural Recommendationcited by 28
  32. 2023An Automated Survey of Generative Artificial Intelligence: Large Language Models, Architectures, Protocols, and Applicationscited by 20
  33. 2021Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Modelscited by 15
  34. 2023Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Tracescited by 13
  35. 2022DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Predictioncited by 12
  36. 2020Cross-Stack Workload Characterization of Deep Recommendation Systemscited by 11
  37. 2023MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendationcited by 9
  38. 2022DeepFlow: A Cross-Stack Pathfinding Framework for Distributed AI Systemscited by 6

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 2; 2026: 4.

14 papers.

Sort:

By area:

  1. 2024Revisiting Reliability in Large-Scale Machine Learning Research Clusterscited by 27
  2. 2024Forecasting GPU Performance for Deep Learning Training and Inferencecited by 15
  3. 2024MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AIcited by 15
  4. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM modelscited by 12
  5. 2024Beyond Efficiency: Scaling AI Sustainablycited by 8
  6. 2024Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Trainingcited by 7
  7. 2025Accelerating Frontier MoE Training with 3D Integrated Opticscited by 1
  8. 2025Scalable Synthesis of distributed LLM workloads through Symbolic Tensor Graphscited by 1
  9. 2023A Survey of Serverless Machine Learning Model Inferencecited by 0
  10. 2024Arena: Efficiently Training Large Models via Dynamic Scheduling and Adaptive Parallelism Co-Designcited by 0
  11. 2026To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Trainingcited by 0
  12. 2026Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Servingcited by 0
2 more
  1. 2026ScaleAcross Explorer: Exploring Communication Optimization for Scale-Across AI Model Trainingcited by 0
  2. 2026Evaluation of ML Resource Utilization Requires Model Life Cycle Assessmentcited by 0