Performance and Power: Systematic Evaluation of AI Workloads on Accelerators with CARAML
Not on the map. Why?
Abstract (arXiv)
The rapid advancement of machine learning (ML) technologies has driven the development of specialized hardware accelerators designed to facilitate more efficient model training. This paper introduces the CARAML benchmark suite, which is employed to assess performance and energy consumption during the training of transformer-based large language models and computer vision models on a range of hardware accelerators, including systems from NVIDIA, AMD, and Graphcore. CARAML provides a compact, automated, extensible, and reproducible framework for assessing the performance and energy of ML workloads across various novel hardware architectures. The design and implementation of CARAML, along with a custom power measurement tool called jpwr, are discussed in detail.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 16 of its 38 references that are in the map. First posted on arXiv 19 Sep 2024.
Builds on
16 of 38 references.
References · 16
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
- 2021Carbon Emissions and Large Neural Network Trainingcited by 567
- 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
- 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
- 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
- 2019MLPerf Inference Benchmarkcited by 199
- 2019MLPerf Training Benchmarkcited by 115
4 more
- 2019Dissecting the Graphcore IPU Architecture via Microbenchmarkingcited by 47
- 2021MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systemscited by 15
- 2023Many Cores, Many Models: GPU Programming Model vs. Vendor Compatibility Overviewcited by 3
- 2024Application-Driven Exascale: The JUPITER Benchmark Suitecited by 3
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.