mlsanityData to 7 Oct 2026

Scaling-laws for Large Time-series Models

arXiv 2405.13867 ↗First posted 22 May 2024cs.LG cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

Scaling laws for large language models (LLMs) have provided useful guidance in training ever larger models for predictable performance gains. Time series forecasting shares a similar sequential structure to language, and is amenable to large-scale transformer architectures. Here we show that foundational decoder-only time series transformer models exhibit analogous scaling-behavior to LLMs, with architectural details (aspect ratio and number of heads) having a minimal effect over broad ranges. We assemble a large corpus of heterogenous time series data on which to train, and establish for the first time power-law scaling with parameter count, dataset size, and training compute, spanning five orders of magnitude.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 40 of its 53 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

40 of 53 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
28 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  3. 2021ViViT: A Video Vision Transformercited by 1,610
  4. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  5. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,004
  6. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 748
  7. 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 744
  8. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  9. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  10. 2024Chronos: Learning the Language of Time Seriescited by 712
  11. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  12. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  13. 2023A decoder-only foundation model for time-series forecastingcited by 650
  14. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
  15. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  16. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  17. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 453
  18. 2018An Empirical Model of Large-Batch Trainingcited by 268
  19. 2021Monash Time Series Forecasting Archivecited by 256
  20. 2023TimeGPT-1cited by 149
  21. 2023LargeST: A Benchmark Dataset for Large-Scale Traffic Forecastingcited by 116
  22. 2020Meta-learning framework with applications to zero-shot time-series forecastingcited by 70
  23. 2023CARD: Channel Aligned Robust Blend Transformer for Time Series Forecastingcited by 55
  24. 2023A Survey on Time-Series Pre-Trained Modelscited by 52
  25. 2018Automatic acoustic detection of birds through deep learning: the first Bird Audio Detection challengecited by 44
  26. 2024SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attentioncited by 43
  27. 2023BuildingsBench: A Large-Scale Dataset of 900K Buildings and Benchmark for Short-Term Load Forecastingcited by 26
  28. 2023Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domaincited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 4.

16 papers.

Sort:

By area:

  1. 2024Scaling Law for Time Series Forecastingcited by 31
  2. 2024Towards Neural Scaling Laws for Time Series Foundation Modelscited by 30
  3. 2025EEG Foundation Models: A Critical Review of Current Progress and Future Directionscited by 30
  4. 2025Rethinking Evaluation in the Era of Time Series Foundation Models: (Un)known Information Leakage Challengescited by 10
  5. 2025Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impactscited by 6
  6. 2025Investigating Compositional Reasoning in Time Series Foundation Modelscited by 3
  7. 2025Evaluating Temporal Plasticity in Foundation Time Series Models for Incremental Fine-tuningcited by 2
  8. 2025Test-Time Efficient Pretrained Model Portfolios for Time Series Forecastingcited by 2
  9. 2026Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddingscited by 2
  10. 2024Domain Fusion Controllable Generalization for Cross-Domain Time Series Forecasting from Multi-Domain Integrated Distributioncited by 1
  11. 2025StarEmbed: Benchmarking Time Series Foundation Models on Astronomical Observations of Variable Starscited by 1
  12. 2025Time Series Foundation Models for Process Model Forecastingcited by 1
4 more
  1. 2025Relative-Based Scaling Law for Neural Language Modelscited by 0
  2. 2026On the Invariance and Generality of Neural Scaling Lawscited by 0
  3. 2026When Do Foundation Models Pay Off? A Break-Even Analysis of Pretrained Time Series Forecasterscited by 0
  4. 2026Learning Spatio-Temporal Foundation Models from Pure Synthetic Datacited by 0