# A Runtime-Based Computational Performance Predictor for Deep Neural Network Training

arXiv: https://arxiv.org/abs/2102.00527 · HTML: https://mlsanity.com/paper/2102.00527/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 90 references that are in the map.
- First posted on arXiv 31 Jan 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [cuDNN: Efficient Primitives for Deep Learning](https://mlsanity.com/paper/1410.0759/) (2014)
- [MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems](https://mlsanity.com/paper/1512.01274/) (2015)
- [Beyond Data and Model Parallelism for Deep Neural Networks](https://mlsanity.com/paper/1807.05358/) (2018)
- [Learning to Optimize Tensor Programs](https://mlsanity.com/paper/1805.08166/) (2018)
- [MLPerf Training Benchmark](https://mlsanity.com/paper/1910.01500/) (2019)
- [Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads](https://mlsanity.com/paper/2008.09213/) (2020)
- [Predicting the Computational Cost of Deep Learning Models](https://mlsanity.com/paper/1811.11880/) (2018)
- [Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks](https://mlsanity.com/paper/1808.07412/) (2018)
- [Asynchrony begets Momentum, with an Application to Deep Learning](https://mlsanity.com/paper/1605.09774/) (2016)
- [Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training](https://mlsanity.com/paper/2006.03318/) (2020)
- [Fathom: Reference Workloads for Modern Deep Learning Methods](https://mlsanity.com/paper/1608.06581/) (2016)
- [Skyline: Interactive In-Editor Computational Performance Profiling for Deep Neural Network Training](https://mlsanity.com/paper/2008.06798/) (2020)
- [Optimizing Memory-Access Patterns for Deep Learning Accelerators](https://mlsanity.com/paper/2002.12798/) (2020)

## Built on by (most cited first, 29 of 29)

- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Decoding Speculative Decoding](https://mlsanity.com/paper/2402.01528/) (2024)
- [Aryl: An Elastic Cluster Scheduler for Deep Learning](https://mlsanity.com/paper/2202.07896/) (2022)
- [Mobile Foundation Model as Firmware](https://mlsanity.com/paper/2308.14363/) (2023)
- [APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving](https://mlsanity.com/paper/2411.17651/) (2024)
- [Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimization](https://mlsanity.com/paper/2503.19050/) (2025)
- [DNNAbacus: Toward Accurate Computational Cost Prediction for Deep Neural Networks](https://mlsanity.com/paper/2205.12095/) (2022)
- [Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing](https://mlsanity.com/paper/2504.19516/) (2025)
- [Efficient Fine-Grained GPU Performance Modeling for Distributed Deep Learning of LLM](https://mlsanity.com/paper/2509.22832/) (2025)
- [CDMPP: A Device-Model Agnostic Framework for Latency Prediction of Tensor Programs](https://mlsanity.com/paper/2311.09690/) (2023)
- [PyGim: An Efficient Graph Neural Network Library for Real Processing-In-Memory Architectures](https://mlsanity.com/paper/2402.16731/) (2024)
- [QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devices](https://mlsanity.com/paper/2407.02327/) (2024)
- [KernelSight-LM: A Kernel-Level LLM Inference Simulator](https://mlsanity.com/paper/2606.28565/) (2026)
- [Inference Latency Prediction at the Edge](https://mlsanity.com/paper/2210.02620/) (2022)
- [DIPPM: a Deep Learning Inference Performance Predictive Model using Graph Neural Networks](https://mlsanity.com/paper/2303.11733/) (2023)
- [Moirai: Towards Optimal Placement for Distributed Inference on Heterogeneous Devices](https://mlsanity.com/paper/2312.04025/) (2023)
- [Cloud-based XAI Services for Assessing Open Repository Models Under Adversarial Attacks](https://mlsanity.com/paper/2401.12261/) (2024)
- [Towards Universal Performance Modeling for Machine Learning Training on Multi-GPU Platforms](https://mlsanity.com/paper/2404.12674/) (2024)
- [PreNeT: Leveraging Computational Features to Predict Deep Neural Network Training Time](https://mlsanity.com/paper/2412.15519/) (2024)
- [Maya: Optimizing Deep Learning Training Workloads using GPU Runtime Emulation](https://mlsanity.com/paper/2503.20191/) (2025)
- [ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusters](https://mlsanity.com/paper/2606.18600/) (2026)
- [Towards providing reliable job completion time predictions using PCS](https://mlsanity.com/paper/2401.10354/) (2024)
- [Rethinking Dynamic Networks and Heterogeneous Computing with Automatic Parallelization](https://mlsanity.com/paper/2506.02787/) (2025)
- [ScaleDL: Towards Scalable and Efficient Runtime Prediction for Distributed Deep Learning Workloads](https://mlsanity.com/paper/2511.04162/) (2025)
- [Load Testing for Machine Learning Model Serving Systems at Scale](https://mlsanity.com/paper/2606.22013/) (2026)
- [SOLAR: AI-Powered Speed-of-Light Performance Analysis](https://mlsanity.com/paper/2606.26383/) (2026)
- [MLSYSIM: First-Principles Infrastructure Modeling for Machine Learning Systems](https://mlsanity.com/paper/2607.02558/) (2026)
- [LLM-Powered Predictive Decision-Making for Sustainable Data Center Operations](https://mlsanity.com/paper/2608.18503/) (2026)
- [ShardMeter: Sharded and Geo-Distributed Training Without the Guesswork](https://mlsanity.com/paper/2608.23840/) (2026)

## Abstract (arXiv)

Deep learning researchers and practitioners usually leverage GPUs to help train their deep neural networks (DNNs) faster. However, choosing which GPU to use is challenging both because (i) there are many options, and (ii) users grapple with competing concerns: maximizing compute performance while minimizing costs. In this work, we present a new practical technique to help users make informed and cost-efficient GPU selections: make performance predictions with the help of a GPU that the user already has. Our technique exploits the observation that, because DNN training consists of repetitive compute steps, predicting the execution time of a single iteration is usually enough to characterize the performance of an entire training process. We make predictions by scaling the execution time of each operation in a training iteration from one GPU to another using either (i) wave scaling, a technique based on a GPU's execution model, or (ii) pre-trained multilayer perceptrons. We implement our technique into a Python library called Habitat and find that it makes accurate iteration execution time predictions (with an average error of 11.8%) on ResNet-50, Inception v3, the Transformer, GNMT, and DCGAN across six different GPU architectures. Habitat supports PyTorch, is easy to use, and is open source.