mlsanityData to 7 Oct 2026

Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts

arXiv 2409.16040 ↗First posted 24 Sep 2024cs.LG cs.AIcited by 280

On the map: Mixture-of-experts → MoE for time series and graphs

Abstract (arXiv)

Deep learning for time series forecasting has seen significant advancements over the past decades. However, despite the success of large-scale pre-training in language and vision domains, pre-trained time series models remain limited in scale and operate at a high cost, hindering the development of larger capable forecasting models in real-world applications. In response, we introduce Time-MoE, a scalable and unified architecture designed to pre-train larger, more capable forecasting foundation models while reducing inference costs. By leveraging a sparse mixture-of-experts (MoE) design, Time-MoE enhances computational efficiency by activating only a subset of networks for each prediction, reducing computational load while maintaining high model capacity. This allows Time-MoE to scale effectively without a corresponding increase in inference costs. Time-MoE comprises a family of decoder-only transformer models that operate in an auto-regressive manner and support flexible forecasting horizons with varying input context lengths. We pre-trained these models on our newly introduced large-scale data Time-300B, which spans over 9 domains and encompassing over 300 billion time points. For the first time, we scaled a time series foundation model up to 2.4 billion parameters, achieving significantly improved forecasting precision. Our results validate the applicability of scaling laws for training tokens and model size in the context of time series forecasting. Compared to dense models with the same number of activated parameters or equivalent computation budgets, our models consistently outperform them by large margin. These advancements position Time-MoE as a state-of-the-art solution for tackling real-world time series forecasting challenges with superior capability, efficiency, and flexibility.

Key facts. Cited by 280 papers in the map (data through 7 Oct 2026). Placed in MoE for time series and graphs, one of 12 directions in Mixture-of-experts. Builds on 73 of its 98 references that are in the map. First posted on arXiv 24 Sep 2024.

Builds on

73 of 98 references.

References · 73

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
38 more
  1. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  2. 2020GLU Variants Improve Transformercited by 1,724
  3. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  5. 2019Root Mean Square Layer Normalizationcited by 1,425
  6. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
  7. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  8. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,030
  9. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,004
  10. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 906
  11. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  12. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  13. 2019Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecastingcited by 748
  14. 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 744
  15. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  16. 2024Chronos: Learning the Language of Time Seriescited by 712
  17. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  18. 2023A decoder-only foundation model for time-series forecastingcited by 650
  19. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
  20. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  21. 2022Transformers in Time Series: A Surveycited by 489
  22. 2021TS2Vec: Towards Universal Representation of Time Seriescited by 463
  23. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 453
  24. 2020A Transformer-based Framework for Multivariate Time Series Representation Learningcited by 449
  25. 2024TimeMixer: Decomposable Multiscale Mixing for Time Series Forecastingcited by 333
  26. 2016Deep Spatio-Temporal Residual Networks for Citywide Crowd Flows Predictioncited by 330
  27. 2023Long-term Forecasting with TiDE: Time-series Dense Encodercited by 327
  28. 2020WeatherBench: A benchmark dataset for data-driven weather forecastingcited by 272
  29. 2021Monash Time Series Forecasting Archivecited by 256
  30. 2022Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistencycited by 246
  31. 2024Foundation Models for Time Series Analysis: A Tutorial and Surveycited by 234
  32. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 198
  33. 2019Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecastingcited by 161
  34. 2023TimeGPT-1cited by 149
  35. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 129
  36. 2023LargeST: A Benchmark Dataset for Large-Scale Traffic Forecastingcited by 116
  37. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  38. 2023DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detectioncited by 103

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 139; 2026: 132.

280 papers.

Sort:

By area:

  1. 2024GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluationcited by 188
  2. 2025Sundial: A Family of Highly Capable Time Series Foundation Modelscited by 168
  3. 2024TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysiscited by 113
  4. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  5. 2024Timer-XL: Long-Context Transformers for Unified Time Series Forecastingcited by 83
  6. 2025This Time is Different: An Observability Perspective on Time Series Foundation Modelscited by 57
  7. 2025MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answeringcited by 50
  8. 2024Towards Neural Scaling Laws for Time Series Foundation Modelscited by 30
  9. 2024TS-Reasoner: Domain-Oriented Time Series Inference Agents for Reasoning and Automated Analysiscited by 28
  10. 2025LightGTS: A Lightweight General Time Series Forecasting Modelcited by 28
  11. 2025Aurora: Towards Universal Generative Multimodal Time Series Forecastingcited by 28
  12. 2025Harnessing Vision Models for Time Series Analysis: A Surveycited by 26
38 more
  1. 2025Kronos: A Foundation Model for the Language of Financial Marketscited by 25
  2. 2025TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Modelscited by 24
  3. 2026Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scalingcited by 24
  4. 2025FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecastingcited by 23
  5. 2025MIRA: Medical Time Series Foundation Model for Real-World Health Datacited by 21
  6. 2025Foundation Models for Time Series: A Surveycited by 19
  7. 2025BLAST: Balanced Sampling Time Series Corpus for Universal Forecasting Modelscited by 16
  8. 2025Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Modelscited by 15
  9. 2025Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learningcited by 14
  10. 2025Multi-Scale Finetuning for Encoder-based Time Series Foundation Modelscited by 13
  11. 2025From Values to Tokens: An LLM-Driven Framework for Context-aware Time Series Forecasting via Symbolic Discretizationcited by 13
  12. 2025SciTS: Scientific Time Series Understanding and Generation with LLMscited by 13
  13. 2025Panda: A pretrained forecast model for chaotic dynamicscited by 12
  14. 2025CauKer: Classification Time Series Foundation Models Can Be Pretrained on Synthetic Datacited by 12
  15. 2025TimeMosaic: Temporal Heterogeneity Guided Time Series Forecasting via Adaptive Granularity Patch and Segment-wise Decodingcited by 12
  16. 2025Evaluating System 1 vs. 2 Reasoning Approaches for Zero-Shot Time Series Forecasting: A Benchmark and Insightscited by 11
  17. 2025VisionTS++: Cross-Modal Time Series Foundation Model with Continual Pre-trained Vision Backbonescited by 11
  18. 2026Distilling Time Series Foundation Models for Efficient Forecastingcited by 11
  19. 2025Learning Soft Sparse Shapes for Efficient Time-Series Classificationcited by 10
  20. 2025FeDaL: Federated Dataset Learning for General Time Series Foundation Modelscited by 10
  21. 2025TS-Reasoner: Aligning Time Series Foundation Models with LLM Reasoningcited by 10
  22. 2025Rethinking Evaluation in the Era of Time Series Foundation Models: (Un)known Information Leakage Challengescited by 10
  23. 2026Revisiting the Generic Transformer: Deconstructing a Strong Baseline for Time Series Foundation Modelscited by 10
  24. 2024A Mamba Foundation Model for Time Series Forecastingcited by 9
  25. 2025XRF V2: A Dataset for Action Summarization with Wi-Fi Signals, and IMUs in Phones, Watches, Earbuds, and Glassescited by 9
  26. 2025Harnessing Vision-Language Models for Time Series Anomaly Detectioncited by 9
  27. 2025FinCast: A Foundation Model for Financial Time-Series Forecastingcited by 9
  28. 2025Towards Foundation Models for Zero-Shot Time Series Anomaly Detection: Leveraging Synthetic Data and Relative Context Discrepancycited by 9
  29. 2025Glocal Information Bottleneck for Time Series Imputationcited by 9
  30. 2025Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE Servingcited by 9
  31. 2024OTIS: Learning High-Quality Time Series Features With Tiny Encoderscited by 8
  32. 2025TimeCopilotcited by 8
  33. 2025Kairos: Toward Adaptive and Parameter-Efficient Time Series Foundation Modelscited by 8
  34. 2025Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibilitycited by 8
  35. 2025Selective Learning for Deep Time Series Forecastingcited by 8
  36. 2026TSAQA: Time Series Analysis Question And Answering Benchmarkcited by 8
  37. 2026MemCast: Memory-Driven Time Series Forecasting with Experience-Conditioned Reasoningcited by 8
  38. 2025DualSG: A Dual-Stream Explicit Semantic-Guided Multivariate Time Series Forecasting Frameworkcited by 7

50 most cited of 280 All 280 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey of Deep Learning for Time Series Forecasting: Architectural Diversity and Open Challenges
  2. 2024Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
  3. 2026Fuzzy-MoE: Interpretable Regime-Conditioned Expert Routing for Non-Stationary Multivariate Time Series Forecasting
  4. 2025Output Scaling: YingLong-Delayed Chain of Thought in a Large Pretrained Time Series Forecasting Model
  5. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Models
  6. 2025Conversational Time Series Foundation Models: Towards Explainable and Effective Forecasting
  7. 2025Wavelet Mixture of Experts for Time Series Forecasting
  8. 2025TimeHF: Billion-Scale Time Series Models Guided by Human Feedback