# MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems arXiv: https://arxiv.org/abs/2110.11466 · HTML: https://mlsanity.com/paper/2110.11466/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Builds on 16 of its 53 references that are in the map. - First posted on arXiv 21 Oct 2021. ## Builds on (most cited first, 16 of 16) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016) - [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016) - [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019) - [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017) - [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019) - [MLPerf Training Benchmark](https://mlsanity.com/paper/1910.01500/) (2019) - [Benchmarking TPU, GPU, and CPU Platforms for Deep Learning](https://mlsanity.com/paper/1907.10701/) (2019) - [Exascale Deep Learning for Climate Analytics](https://mlsanity.com/paper/1810.01993/) (2018) - [Fathom: Reference Workloads for Modern Deep Learning Methods](https://mlsanity.com/paper/1608.06581/) (2016) - [Machine Learning and Big Scientific Data](https://mlsanity.com/paper/1910.07631/) (2019) - [CosmoFlow: Using Deep Learning to Learn the Universe at Scale](https://mlsanity.com/paper/1808.04728/) (2018) - [Exploiting Parallelism Opportunities with Deep Learning Frameworks](https://mlsanity.com/paper/1908.04705/) (2019) - [A Modular Benchmarking Infrastructure for High-Performance and Reproducible Deep Learning](https://mlsanity.com/paper/1901.10183/) (2019) - [XSP: Across-Stack Profiling and Analysis of Machine Learning Models on GPUs](https://mlsanity.com/paper/1908.06869/) (2019) ## Built on by (most cited first, 15 of 15) - [MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI](https://mlsanity.com/paper/2410.12032/) (2024) - [XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaverse](https://mlsanity.com/paper/2211.08675/) (2022) - [FastML Science Benchmarks: Accelerating Real-Time Scientific Edge Machine Learning](https://mlsanity.com/paper/2207.07958/) (2022) - [ScaleFold: Reducing AlphaFold Initial Training Time to 10 Hours](https://mlsanity.com/paper/2404.11068/) (2024) - [Breaking Boundaries: Distributed Domain Decomposition with Scalable Physics-Informed Neural PDE Solvers](https://mlsanity.com/paper/2308.14258/) (2023) - [I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey](https://mlsanity.com/paper/2404.10386/) (2024) - [AI Benchmark Democratization and Carpentry](https://mlsanity.com/paper/2512.11588/) (2025) - [Performance and Power: Systematic Evaluation of AI Workloads on Accelerators with CARAML](https://mlsanity.com/paper/2409.12994/) (2024) - [An MLCommons Scientific Benchmarks Ontology](https://mlsanity.com/paper/2511.05614/) (2025) - [An Overview of MLCommons Cloud Mask Benchmark: Related Research and Data](https://mlsanity.com/paper/2312.04799/) (2023) - [SOLAR: A Highly Optimized Data Loading Framework for Distributed Training of CNN-based Scientific Surrogates](https://mlsanity.com/paper/2211.00224/) (2022) - [SAIH: A Scalable Evaluation Methodology for Understanding AI Performance Trend on HPC Systems](https://mlsanity.com/paper/2212.03410/) (2022) - [Improvements & Evaluations on the MLCommons CloudMask Benchmark](https://mlsanity.com/paper/2403.04553/) (2024) - [Introducing Milabench: Benchmarking Accelerators for AI](https://mlsanity.com/paper/2411.11940/) (2024) - [Hybrid Learning and Optimization-Based Dynamic Scheduling for DL Workloads on Heterogeneous GPU Clusters](https://mlsanity.com/paper/2512.10271/) (2025) ## Abstract (arXiv) Scientific communities are increasingly adopting machine learning and deep learning models in their applications to accelerate scientific insights. High performance computing systems are pushing the frontiers of performance with a rich diversity of hardware resources and massive scale-out capabilities. There is a critical need to understand fair and effective benchmarking of machine learning applications that are representative of real-world scientific use cases. MLPerf is a community-driven standard to benchmark machine learning workloads, focusing on end-to-end performance metrics. In this paper, we introduce MLPerf HPC, a benchmark suite of large-scale scientific machine learning training applications driven by the MLCommons Association. We present the results from the first submission round, including a diverse set of some of the world's largest HPC systems. We develop a systematic framework for their joint analysis and compare them in terms of data staging, algorithmic convergence, and compute performance. As a result, we gain a quantitative understanding of optimizations on different subsystems such as staging and on-node loading of data, compute-unit utilization, and communication scheduling, enabling overall >10 × (end-to-end) performance improvements through system scaling. Notably, our analysis shows a scale-dependent interplay between the dataset size, a system's memory hierarchy, and training convergence that underlines the importance of near-compute storage. To overcome the data-parallel scalability challenge at large batch sizes, we discuss specific learning techniques and hybrid data-and-model parallelism that are effective on large systems. We conclude by characterizing each benchmark with respect to low-level memory, I/O, and network behavior to parameterize extended roofline performance models in future rounds.