# Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scaling

arXiv: https://arxiv.org/abs/2603.04791 · HTML: https://mlsanity.com/paper/2603.04791/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in MoE for time series and graphs, one of 12 directions in Mixture-of-experts.
- Builds on 43 of its 61 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Placed in

- Mixture-of-experts / [MoE for time series and graphs](https://mlsanity.com/moe/moe-for-time-series-and-graphs/)

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [Time Series Forecasting With Deep Learning: A Survey](https://mlsanity.com/paper/2004.13408/) (2020)
- [Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting](https://mlsanity.com/paper/2205.14415/) (2022)
- [Better & Faster Large Language Models via Multi-token Prediction](https://mlsanity.com/paper/2404.19737/) (2024)
- [Blockwise Parallel Decoding for Deep Autoregressive Models](https://mlsanity.com/paper/1811.03115/) (2018)
- [Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts](https://mlsanity.com/paper/2409.16040/) (2024)
- [Query-Key Normalization for Transformers](https://mlsanity.com/paper/2010.04245/) (2020)
- [GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation](https://mlsanity.com/paper/2410.10393/) (2024)
- [Timer: Generative Pre-trained Transformers Are Large Time Series Models](https://mlsanity.com/paper/2402.02368/) (2024)
- [Deep Time Series Models: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2407.13278/) (2024)
- [Chronos-2: From Univariate to Universal Forecasting](https://mlsanity.com/paper/2510.15821/) (2025)
- [Sundial: A Family of Highly Capable Time Series Foundation Models](https://mlsanity.com/paper/2502.00816/) (2025)
- [TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning](https://mlsanity.com/paper/2505.23719/) (2025)
- [ForecastPFN: Synthetically-Trained Zero-Shot Forecasting](https://mlsanity.com/paper/2311.01933/) (2023)
- [Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts](https://mlsanity.com/paper/2410.10469/) (2024)
- [Timer-XL: Long-Context Transformers for Unified Time Series Forecasting](https://mlsanity.com/paper/2410.04803/) (2024)
- [Moirai 2.0: When Less Is More for Time Series Forecasting](https://mlsanity.com/paper/2511.11698/) (2025)

## Built on by (most cited first, 28 of 28)

- [TiRex-2: Generalizing TiRex to Multivariate Data and Streaming](https://mlsanity.com/paper/2607.01204/) (2026)
- [Falcon-X: A Time Series Foundation Model for Heterogeneous Multivariate Modeling](https://mlsanity.com/paper/2605.27286/) (2026)
- [Learning the Context of Errors: Black-Box Online Adaptation of Time Series Foundation Models](https://mlsanity.com/paper/2606.14222/) (2026)
- [Reasoning-Aware Training for Time Series Forecasting](https://mlsanity.com/paper/2605.08625/) (2026)
- [Into the ORBIT for Time Series: Training Regimes for Foundation Models](https://mlsanity.com/paper/2608.13262/) (2026)
- [Beyond Numerical Time Series: A Unified Benchmark for Multimodal Forecasting with Heterogeneous Context](https://mlsanity.com/paper/2609.15087/) (2026)
- [t0: A Time-Series Foundation Model for Forecasting with Context](https://mlsanity.com/paper/2609.24559/) (2026)
- [Don't Learn the Shape: Forecasting Periodic Time Series by Rank-1 Decomposition](https://mlsanity.com/paper/2605.07222/) (2026)
- [Factorize to Generalize: Retrieval-Guided Invariant-Dynamic Decomposition for Time Series Forecasting](https://mlsanity.com/paper/2605.24911/) (2026)
- [TSQAgent: Rating Time Series Data Quality via Dedicated Agentic Reasoning](https://mlsanity.com/paper/2606.03629/) (2026)
- [LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications](https://mlsanity.com/paper/2608.23058/) (2026)
- [Tabby: An Open Pretraining Recipe for Time Series Foundation Models](https://mlsanity.com/paper/2609.13956/) (2026)
- [Tyan-WP: A Wind Power Foundation Model for Ultra-Short-Term Probabilistic Forecasting](https://mlsanity.com/paper/2606.08630/) (2026)
- [TimeVista: Exploring and Exploiting Vision-Language Models as Judges for Time Series Forecasting](https://mlsanity.com/paper/2606.16173/) (2026)
- [AgenticTwin: An Agentic LLM Framework Integrated with Digital Twin for Anomaly Detection](https://mlsanity.com/paper/2608.11679/) (2026)
- [One Patch, Three Roles: What Is Actually Coupled in Autoregressive Time-Series Forecasting?](https://mlsanity.com/paper/2609.23686/) (2026)
- [SGA: Uncertainty Quantification for Multi-Step Forecasting in Time Series Foundation Models](https://mlsanity.com/paper/2609.28582/) (2026)
- [TimeBraid: Unifying Time Series and Language for Understanding and Forecasting](https://mlsanity.com/paper/2609.29792/) (2026)
- [Aurora-X: Built for Extreme Time Series Forecasting](https://mlsanity.com/paper/2609.31038/) (2026)
- [Self-Evolving Time-Series Forecasting Agents with Episodic Memory and Online Policy Learning](https://mlsanity.com/paper/2609.32689/) (2026)
- [Instance-Adaptive Prompts as Context for Time-Series Foundation Models](https://mlsanity.com/paper/2609.34786/) (2026)
- [Raw-Routed Mixture of Adapters: A Causal Intervention for Routing Collapse in Time Series Foundation Models](https://mlsanity.com/paper/2609.39445/) (2026)
- [OpenTSLM TeeMoE: A Unified Time-Series Language Model for Forecasting, Contextual Prediction, and Reasoning](https://mlsanity.com/paper/2609.40265/) (2026)
- [Pythia: Toward Foundation World Models for Multimodal Time Series](https://mlsanity.com/paper/2610.05240/) (2026)
- [A Unified Scaling Law for Time Series Foundation Models](https://mlsanity.com/paper/2610.05269/) (2026)
- [RideBench: A Large-Scale Exogenous-Aware Benchmark for Ride-Hailing Time Series Forecasting](https://mlsanity.com/paper/2610.11164/) (2026)
- [Timer-M1: A Multivariate Time Series Foundation Model via Learning Primitives](https://mlsanity.com/paper/2610.11734/) (2026)
- [ECHO: Correcting Time Series Foundation Model Forecasts with Historical Analogues and Errors](https://mlsanity.com/paper/2610.16433/) (2026)

## Abstract (arXiv)

We introduce Timer-S1, a strong Mixture-of-Experts (MoE) time series foundation model with 8.3B total parameters, 0.75B activated parameters for each token, and a context length of 11.5K. To overcome the scalability bottleneck in existing pre-trained time series foundation models, we perform Serial Scaling in three dimensions: model architecture, dataset, and training pipeline. Timer-S1 integrates sparse TimeMoE blocks and generic TimeSTP blocks for Serial-Token Prediction (STP), a generic training objective that adheres to the serial nature of forecasting. The proposed paradigm introduces serial computations to improve long-term predictions while avoiding costly rolling-style inference and pronounced error accumulation in the standard next-token prediction. Pursuing a high-quality and unbiased training dataset, we curate TimeBench, a corpus with one trillion time points, and apply meticulous data augmentation to mitigate predictive bias. We further pioneer a post-training stage, including continued pre-training and long-context extension, to enhance short-term and long-context performance. Evaluated on the large-scale GIFT-Eval leaderboard, Timer-S1 achieves state-of-the-art forecasting performance, attaining the best MASE and CRPS scores as a pre-trained model. Timer-S1 is released to facilitate further research.