# Beyond Human-Level Accuracy: Computational Challenges in Deep Learning

arXiv: https://arxiv.org/abs/1909.01736 · HTML: https://mlsanity.com/paper/1909.01736/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2019.

## Built on by (most cited first, 19 of 19)

- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws](https://mlsanity.com/paper/2401.00448/) (2023)
- [Language models scale reliably with over-training and on downstream tasks](https://mlsanity.com/paper/2403.08540/) (2024)
- [ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Model](https://mlsanity.com/paper/2412.14559/) (2024)
- [Understanding Scaling Laws for Recommendation Models](https://mlsanity.com/paper/2208.08489/) (2022)
- [Scaling MLPs: A Tale of Inductive Bias](https://mlsanity.com/paper/2306.13575/) (2023)
- [Pre-training under infinite compute](https://mlsanity.com/paper/2509.14786/) (2025)
- [Scaling Laws for Upcycling Mixture-of-Experts Language Models](https://mlsanity.com/paper/2502.03009/) (2025)
- [Requirements Engineering for Machine Learning: A Review and Reflection](https://mlsanity.com/paper/2210.00859/) (2022)
- [Reranking Laws for Language Generation: A Communication-Theoretic Perspective](https://mlsanity.com/paper/2409.07131/) (2024)
- [Is the Number of Trainable Parameters All That Actually Matters?](https://mlsanity.com/paper/2109.11928/) (2021)
- [Edge-assisted U-Shaped Split Federated Learning with Privacy-preserving for Internet of Things](https://mlsanity.com/paper/2311.04944/) (2023)
- [Reusing Overtrained Language Models Saturates Scaling](https://mlsanity.com/paper/2510.06548/) (2025)
- [Navigating causal deep learning](https://mlsanity.com/paper/2212.00911/) (2022)
- [Analyzing and reducing the synthetic-to-real transfer gap in Music Information Retrieval: the task of automatic drum transcription](https://mlsanity.com/paper/2407.19823/) (2024)
- [Cross-Subject Deep Transfer Models for Evoked Potentials in Brain-Computer Interface](https://mlsanity.com/paper/2301.12322/) (2023)
- [Hera: A Heterogeneity-Aware Multi-Tenant Inference Server for Personalized Recommendations](https://mlsanity.com/paper/2302.11750/) (2023)
- [The Role of Entanglement in Quantum Reservoir Computing with Coupled Kerr Nonlinear Oscillators](https://mlsanity.com/paper/2508.11175/) (2025)

## Abstract (arXiv)

Deep learning (DL) research yields accuracy and product improvements from both model architecture changes and scale: larger data sets and models, and more computation. For hardware design, it is difficult to predict DL model changes. However, recent prior work shows that as dataset sizes grow, DL model accuracy and model size grow predictably. This paper leverages the prior work to project the dataset and model size growth required to advance DL accuracy beyond human-level, to frontier targets defined by machine learning experts. Datasets will need to grow 33--971 ×, while models will need to grow 6.6--456× to achieve target accuracies. We further characterize and project the computational requirements to train these applications at scale. Our characterization reveals an important segmentation of DL training challenges for recurrent neural networks (RNNs) that contrasts with prior studies of deep convolutional networks. RNNs will have comparatively moderate operational intensities and very large memory footprint requirements. In contrast to emerging accelerator designs, large-scale RNN training characteristics suggest designs with significantly larger memory capacity and on-chip caches.