# Towards Neural Scaling Laws for Time Series Foundation Models

arXiv: https://arxiv.org/abs/2410.12360 · HTML: https://mlsanity.com/paper/2410.12360/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 54 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Builds on (most cited first, 30 of 43)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis](https://mlsanity.com/paper/2210.02186/) (2022)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [Jamba: A Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2403.19887/) (2024)
- [Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts](https://mlsanity.com/paper/2409.16040/) (2024)
- [TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods](https://mlsanity.com/paper/2403.20150/) (2024)
- [Timer: Generative Pre-trained Transformers Are Large Time Series Models](https://mlsanity.com/paper/2402.02368/) (2024)
- [Deep Time Series Models: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2407.13278/) (2024)
- [Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance](https://mlsanity.com/paper/2403.16952/) (2024)
- [Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws](https://mlsanity.com/paper/2401.00448/) (2023)
- [LargeST: A Benchmark Dataset for Large-Scale Traffic Forecasting](https://mlsanity.com/paper/2306.08259/) (2023)
- [A Dynamical Model of Neural Scaling Laws](https://mlsanity.com/paper/2402.01092/) (2024)
- [Scaling Laws for Neural Machine Translation](https://mlsanity.com/paper/2109.07740/) (2021)
- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](https://mlsanity.com/paper/2310.10196/) (2023)
- [Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects](https://mlsanity.com/paper/2306.10125/) (2023)

## Built on by (most cited first, 30 of 33)

- [This Time is Different: An Observability Perspective on Time Series Foundation Models](https://mlsanity.com/paper/2505.14766/) (2025)
- [EEG Foundation Models: A Critical Review of Current Progress and Future Directions](https://mlsanity.com/paper/2507.11783/) (2025)
- [Aurora: Towards Universal Generative Multimodal Time Series Forecasting](https://mlsanity.com/paper/2509.22295/) (2025)
- [TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Models](https://mlsanity.com/paper/2509.24803/) (2025)
- [Kronos: A Foundation Model for the Language of Financial Markets](https://mlsanity.com/paper/2508.02739/) (2025)
- [Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learning](https://mlsanity.com/paper/2505.11349/) (2025)
- [Out-of-Distribution Generalization in Time Series: A Survey](https://mlsanity.com/paper/2503.13868/) (2025)
- [CauKer: Classification Time Series Foundation Models Can Be Pretrained on Synthetic Data](https://mlsanity.com/paper/2508.02879/) (2025)
- [FeDaL: Federated Dataset Learning for General Time Series Foundation Models](https://mlsanity.com/paper/2508.04045/) (2025)
- [There are no Champions in Supervised Long-Term Time Series Forecasting](https://mlsanity.com/paper/2502.14045/) (2025)
- [Xihe: Scalable Zero-Shot Time Series Learner Via Hierarchical Interleaved Block Attention](https://mlsanity.com/paper/2510.21795/) (2025)
- [Position: Universal Time Series Foundation Models Rest on a Category Error](https://mlsanity.com/paper/2602.05287/) (2026)
- [Investigating Compositional Reasoning in Time Series Foundation Models](https://mlsanity.com/paper/2502.06037/) (2025)
- [From Tokenizer Bias to Backbone Capability: A Controlled Study of LLMs for Time Series Forecasting](https://mlsanity.com/paper/2504.08818/) (2025)
- [A Theoretical Analysis of Detecting Large Model-Generated Time Series](https://mlsanity.com/paper/2511.07104/) (2025)
- [TS-Arena -- A Live Forecast Pre-Registration Platform](https://mlsanity.com/paper/2512.20761/) (2025)
- [Diversified Scaling Inference in Time Series Foundation Models](https://mlsanity.com/paper/2601.17376/) (2026)
- [OATS: Online Data Augmentation for Time Series Foundation Models](https://mlsanity.com/paper/2601.19040/) (2026)
- [Mitigating Data Scarcity in Time Series Analysis: A Foundation Model with Series-Symbol Data Generation](https://mlsanity.com/paper/2502.15466/) (2025)
- [A Comparative Study on How Data Normalization Affects Zero-Shot Generalization in Time Series Foundation Models](https://mlsanity.com/paper/2512.02833/) (2025)
- [EIDOS: Latent-Space Predictive Learning for Time Series Foundation Models](https://mlsanity.com/paper/2602.14024/) (2026)
- [Discrete Prototypical Memories for Federated Time Series Foundation Models](https://mlsanity.com/paper/2604.04475/) (2026)
- [WaveMoE: A Wavelet-Enhanced Mixture-of-Experts Foundation Model for Time Series Forecasting](https://mlsanity.com/paper/2604.10544/) (2026)
- [MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification](https://mlsanity.com/paper/2605.11408/) (2026)
- [How Do Electrocardiogram Models Scale?](https://mlsanity.com/paper/2605.17276/) (2026)
- [Towards Multi-Fidelity Scaling Laws of Neural Surrogates in CFD](https://mlsanity.com/paper/2511.01830/) (2025)
- [Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation](https://mlsanity.com/paper/2605.08541/) (2026)
- [GeoGNN: Time Series Geo-Localization using Two-Tower Graph Neural Networks](https://mlsanity.com/paper/2606.08303/) (2026)
- [Learning Spatio-Temporal Foundation Models from Pure Synthetic Data](https://mlsanity.com/paper/2607.16251/) (2026)
- [Scale-Aware Pretraining of Time Series Foundation Models via Multi-Patch Token Alignment and Hybrid Masking](https://mlsanity.com/paper/2608.20005/) (2026)

## Abstract (arXiv)

Scaling laws offer valuable insights into the design of time series foundation models (TSFMs). However, previous research has largely focused on the scaling laws of TSFMs for in-distribution (ID) data, leaving their out-of-distribution (OOD) scaling behavior and the influence of model architectures less explored. In this work, we examine two common TSFM architectures, encoder-only and decoder-only Transformers, and investigate their scaling behavior on both ID and OOD data. These models are trained and evaluated across varying parameter counts, compute budgets, and dataset sizes. Our experiments reveal that the log-likelihood loss of TSFMs exhibits similar scaling behavior in both OOD and ID settings. We further compare the scaling properties across different architectures, incorporating two state-of-the-art TSFMs as case studies, showing that model architecture plays a significant role in scaling. The encoder-only Transformers demonstrate better scalability than the decoder-only Transformers, while the architectural enhancements in the two advanced TSFMs primarily improve ID performance but reduce OOD scalability. While scaling up TSFMs is expected to drive performance breakthroughs, the lack of a comprehensive understanding of TSFM scaling laws has hindered the development of a robust framework to guide model scaling. We fill this gap in this work by synthesizing our findings and providing practical guidelines for designing and scaling larger TSFMs with enhanced model capabilities.