mlsanityData to 14 Oct 2026

Introducing Milabench: Benchmarking Accelerators for AI

arXiv 2411.11940 ↗First posted 18 Nov 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

AI workloads, particularly those driven by deep learning, are introducing novel usage patterns to high-performance computing (HPC) systems that are not comprehensively captured by standard HPC benchmarks. As one of the largest academic research centers dedicated to deep learning, Mila identified the need to develop a custom benchmarking suite to address the diverse requirements of its community, which consists of over 1,000 researchers. This report introduces Milabench, the resulting benchmarking suite. Its design was informed by an extensive literature review encompassing 867 papers, as well as surveys conducted with Mila researchers. This rigorous process led to the selection of 26 primary benchmarks tailored for procurement evaluations, alongside 16 optional benchmarks for in-depth analysis. We detail the design methodology, the structure of the benchmarking suite, and provide performance evaluations using GPUs from NVIDIA, AMD, and Intel. The Milabench suite is open source and can be accessed at github.com/mila-iqia/milabench.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 28 references that are in the map. First posted on arXiv 18 Nov 2024.

Builds on

20 of 28 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2022A ConvNet for the 2020scited by 3,579
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2017The Kinetics Human Action Video Datasetcited by 1,909
  11. 2020Reformer: The Efficient Transformercited by 1,668
  12. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
8 more
  1. 2020Designing Network Design Spacescited by 960
  2. 2020Directional Message Passing for Molecular Graphscited by 561
  3. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  4. 2020Principal Neighbourhood Aggregation for Graph Netscited by 478
  5. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  6. 2019MLPerf Inference Benchmarkcited by 199
  7. 2021MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systemscited by 15
  8. 2024Application-Driven Exascale: The JUPITER Benchmark Suitecited by 3

Built on by

Not cited yet.