# Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark

arXiv: https://arxiv.org/abs/1806.01427 · HTML: https://mlsanity.com/paper/1806.01427/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2018.

## Built on by (most cited first, 21 of 21)

- [Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision](https://mlsanity.com/paper/2109.08203/) (2021)
- [Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training](https://mlsanity.com/paper/2208.06102/) (2022)
- [Codabench: Flexible, Easy-to-Use and Reproducible Benchmarking Platform](https://mlsanity.com/paper/2110.05802/) (2021)
- [Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads](https://mlsanity.com/paper/2105.11118/) (2021)
- [Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs](https://mlsanity.com/paper/2204.12013/) (2022)
- [FFCV: Accelerating Training by Removing Data Bottlenecks](https://mlsanity.com/paper/2306.12517/) (2023)
- [APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference](https://mlsanity.com/paper/2401.12200/) (2024)
- [On the Utility of Gradient Compression in Distributed Training Systems](https://mlsanity.com/paper/2103.00543/) (2021)
- [FedAdapter: Efficient Federated Learning for Modern NLP](https://mlsanity.com/paper/2205.10162/) (2022)
- [Knowledge Graph Embedding Methods for Entity Alignment: An Experimental Review](https://mlsanity.com/paper/2203.09280/) (2022)
- [Sync-Switch: Hybrid Parameter Synchronization for Distributed Deep Learning](https://mlsanity.com/paper/2104.08364/) (2021)
- [Deep Learning Inference Frameworks Benchmark](https://mlsanity.com/paper/2210.04323/) (2022)
- [Chasing Low-Carbon Electricity for Practical and Sustainable DNN Training](https://mlsanity.com/paper/2303.02508/) (2023)
- [Neural Loss Function Evolution for Large-Scale Image Classifier Convolutional Neural Networks](https://mlsanity.com/paper/2403.08793/) (2024)
- [An Overview of the Data-Loader Landscape: Comparative Performance Analysis](https://mlsanity.com/paper/2209.13705/) (2022)
- [OrionBench: Benchmarking Time Series Generative Models in the Service of the End-User](https://mlsanity.com/paper/2310.17748/) (2023)
- [Informed Machine Learning for Improved Similarity Assessment in Process-Oriented Case-Based Reasoning](https://mlsanity.com/paper/2106.15931/) (2021)
- [Bioinspired Cortex-based Fast Codebook Generation](https://mlsanity.com/paper/2201.12322/) (2022)
- [SSI-GAN: Semi-Supervised Swin-Inspired Generative Adversarial Networks for Neuronal Spike Classification](https://mlsanity.com/paper/2601.00189/) (2026)
- [Curvature-aware dynamic precision approach for physics-informed neural networks](https://mlsanity.com/paper/2606.04736/) (2026)
- [Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference](https://mlsanity.com/paper/2609.05275/) (2026)

## Abstract (arXiv)

Researchers have proposed hardware, software, and algorithmic optimizations to improve the computational performance of deep learning. While some of these optimizations perform the same operations faster (e.g., increasing GPU clock speed), many others modify the semantics of the training procedure (e.g., reduced precision), and can impact the final model's accuracy on unseen data. Due to a lack of standard evaluation criteria that considers these trade-offs, it is difficult to directly compare these optimizations. To address this problem, we recently introduced DAWNBench, a benchmark competition focused on end-to-end training time to achieve near-state-of-the-art accuracy on an unseen dataset---a combined metric called time-to-accuracy (TTA). In this work, we analyze the entries from DAWNBench, which received optimized submissions from multiple industrial groups, to investigate the behavior of TTA as a metric as well as trends in the best-performing entries. We show that TTA has a low coefficient of variation and that models optimized for TTA generalize nearly as well as those trained using standard methods. Additionally, even though DAWNBench entries were able to train ImageNet models in under 3 minutes, we find they still underutilize hardware capabilities such as Tensor Cores. Furthermore, we find that distributed entries can spend more than half of their time on communication. We show similar findings with entries to the MLPERF v0.5 benchmark.