# Time Matters: Scaling Laws for Any Budget

arXiv: https://arxiv.org/abs/2406.18922 · HTML: https://mlsanity.com/paper/2406.18922/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 5 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 4 of 4)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Chinchilla Scaling: A replication attempt](https://mlsanity.com/paper/2404.10102/) (2024)

## Built on by (most cited first, 4 of 4)

- [Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training](https://mlsanity.com/paper/2505.13738/) (2025)
- [A Hitchhiker's Guide to Scaling Law Estimation](https://mlsanity.com/paper/2410.11840/) (2024)
- [Gemstones: A Model Suite for Multi-Faceted Scaling Laws](https://mlsanity.com/paper/2502.06857/) (2025)
- [Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRI](https://mlsanity.com/paper/2606.26894/) (2026)

## Abstract (arXiv)

A primary cost driver for training large models is wall-clock training time. We show that popular time estimates based on FLOPs are poor estimates, and construct a more accurate proxy based on memory copies. This allows us to accurately estimate the training speed of a transformer model from its hyperparameters. Combined with a scaling law curve like Chinchilla, this allows us to accurately predict the final loss of a model from a simple equation. We show that this expression is accurate across a wide range of model hyperparameter values, enabling us to analytically make architectural decisions and train models more efficiently. Crucially, this analysis predicts that in contrast to existing literature, models should be wider rather than deeper, as the benefits of speed outweigh the benefits of depth.