# Scaling-laws for Large Time-series Models

arXiv: https://arxiv.org/abs/2405.13867 · HTML: https://mlsanity.com/paper/2405.13867/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 53 references that are in the map.
- First posted on arXiv 22 May 2024.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [An Empirical Model of Large-Batch Training](https://mlsanity.com/paper/1812.06162/) (2018)

## Built on by (most cited first, 18 of 18)

- [Scaling Law for Time Series Forecasting](https://mlsanity.com/paper/2405.15124/) (2024)
- [Towards Neural Scaling Laws for Time Series Foundation Models](https://mlsanity.com/paper/2410.12360/) (2024)
- [EEG Foundation Models: A Critical Review of Current Progress and Future Directions](https://mlsanity.com/paper/2507.11783/) (2025)
- [Rethinking Evaluation in the Era of Time Series Foundation Models: (Un)known Information Leakage Challenges](https://mlsanity.com/paper/2510.13654/) (2025)
- [Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts](https://mlsanity.com/paper/2507.19477/) (2025)
- [Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings](https://mlsanity.com/paper/2605.31580/) (2026)
- [Investigating Compositional Reasoning in Time Series Foundation Models](https://mlsanity.com/paper/2502.06037/) (2025)
- [Evaluating Temporal Plasticity in Foundation Time Series Models for Incremental Fine-tuning](https://mlsanity.com/paper/2504.14677/) (2025)
- [Test-Time Efficient Pretrained Model Portfolios for Time Series Forecasting](https://mlsanity.com/paper/2510.06419/) (2025)
- [Domain Fusion Controllable Generalization for Cross-Domain Time Series Forecasting from Multi-Domain Integrated Distribution](https://mlsanity.com/paper/2412.03068/) (2024)
- [StarEmbed: Benchmarking Time Series Foundation Models on Astronomical Observations of Variable Stars](https://mlsanity.com/paper/2510.06200/) (2025)
- [Relative-Based Scaling Law for Neural Language Models](https://mlsanity.com/paper/2510.20387/) (2025)
- [Time Series Foundation Models for Process Model Forecasting](https://mlsanity.com/paper/2512.07624/) (2025)
- [On the Invariance and Generality of Neural Scaling Laws](https://mlsanity.com/paper/2605.07546/) (2026)
- [Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation](https://mlsanity.com/paper/2605.08541/) (2026)
- [When Do Foundation Models Pay Off? A Break-Even Analysis of Pretrained Time Series Forecasters](https://mlsanity.com/paper/2607.04919/) (2026)
- [Learning Spatio-Temporal Foundation Models from Pure Synthetic Data](https://mlsanity.com/paper/2607.16251/) (2026)
- [A Unified Scaling Law for Time Series Foundation Models](https://mlsanity.com/paper/2610.05269/) (2026)

## Abstract (arXiv)

Scaling laws for large language models (LLMs) have provided useful guidance in training ever larger models for predictable performance gains. Time series forecasting shares a similar sequential structure to language, and is amenable to large-scale transformer architectures. Here we show that foundational decoder-only time series transformer models exhibit analogous scaling-behavior to LLMs, with architectural details (aspect ratio and number of heads) having a minimal effect over broad ranges. We assemble a large corpus of heterogenous time series data on which to train, and establish for the first time power-law scaling with parameter count, dataset size, and training compute, spanning five orders of magnitude.