# Predicting Training Time Without Training

arXiv: https://arxiv.org/abs/2008.12478 · HTML: https://mlsanity.com/paper/2008.12478/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2020.

## Built on by (most cited first, 22 of 22)

- [Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models](https://mlsanity.com/paper/2305.12827/) (2023)
- [What can linearized neural networks actually say about generalization?](https://mlsanity.com/paper/2106.06770/) (2021)
- [A linearized framework and a new benchmark for model selection for fine-tuning](https://mlsanity.com/paper/2102.00084/) (2021)
- [Scaling Neural Tangent Kernels via Sketching and Random Features](https://mlsanity.com/paper/2106.07880/) (2021)
- [TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernels](https://mlsanity.com/paper/2207.06343/) (2022)
- [Diffusion Soup: Model Merging for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2406.08431/) (2024)
- [Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?](https://mlsanity.com/paper/2203.14577/) (2022)
- [Estimating informativeness of samples with Smooth Unique Information](https://mlsanity.com/paper/2101.06640/) (2021)
- [Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation](https://mlsanity.com/paper/2302.01428/) (2023)
- [Neural Lineage](https://mlsanity.com/paper/2406.11129/) (2024)
- [Benchmarking Resource Usage for Efficient Distributed Deep Learning](https://mlsanity.com/paper/2201.12423/) (2022)
- [Random Features for the Neural Tangent Kernel](https://mlsanity.com/paper/2104.01351/) (2021)
- [Efficient Fine-Grained GPU Performance Modeling for Distributed Deep Learning of LLM](https://mlsanity.com/paper/2509.22832/) (2025)
- [Cold Posteriors through PAC-Bayes](https://mlsanity.com/paper/2206.11173/) (2022)
- [The fine print on tempered posteriors](https://mlsanity.com/paper/2309.05292/) (2023)
- [Toward a Holistic Approach to Continual Model Merging](https://mlsanity.com/paper/2509.23592/) (2025)
- [Predicting the Encoding Error of SIRENs](https://mlsanity.com/paper/2410.21645/) (2024)
- [The Underlying Correlated Dynamics in Neural Training](https://mlsanity.com/paper/2212.09040/) (2022)
- [PreNeT: Leveraging Computational Features to Predict Deep Neural Network Training Time](https://mlsanity.com/paper/2412.15519/) (2024)
- [Off-the-Shelf Neural Network Architectures for Forex Time Series Prediction come at a Cost](https://mlsanity.com/paper/2405.10679/) (2024)
- [UnifiedNN: Efficient Neural Network Training on the Cloud](https://mlsanity.com/paper/2408.01331/) (2024)
- [DecTrain: Deciding When to Train a Monocular Depth DNN Online](https://mlsanity.com/paper/2410.02980/) (2024)

## Abstract (arXiv)

We tackle the problem of predicting the number of optimization steps that a pre-trained deep network needs to converge to a given value of the loss function. To do so, we leverage the fact that the training dynamics of a deep network during fine-tuning are well approximated by those of a linearized model. This allows us to approximate the training loss and accuracy at any point during training by solving a low-dimensional Stochastic Differential Equation (SDE) in function space. Using this result, we are able to predict the time it takes for Stochastic Gradient Descent (SGD) to fine-tune a model to a given loss without having to perform any training. In our experiments, we are able to predict training time of a ResNet within a 20% error margin on a variety of datasets and hyper-parameters, at a 30 to 45-fold reduction in cost compared to actual training. We also discuss how to further reduce the computational and memory cost of our method, and in particular we show that by exploiting the spectral properties of the gradients' matrix it is possible predict training time on a large dataset while processing only a subset of the samples.