mlsanityData to 13 Oct 2026

Unified Training of Universal Time Series Forecasting Transformers

arXiv 2402.02592 ↗First posted 4 Feb 2024cs.LG cs.AIcited by 602

Not on the map. Why?

Abstract (arXiv)

Deep learning for time series forecasting has traditionally operated within a one-model-per-dataset framework, limiting its potential to leverage the game-changing impact of large pre-trained models. The concept of universal forecasting, emerging from pre-training on a vast collection of time series datasets, envisions a single Large Time Series Model capable of addressing diverse downstream forecasting tasks. However, constructing such a model poses unique challenges specific to time series data: i) cross-frequency learning, ii) accommodating an arbitrary number of variates for multivariate time series, and iii) addressing the varying distributional properties inherent in large-scale data. To address these challenges, we present novel enhancements to the conventional time series Transformer architecture, resulting in our proposed Masked Encoder-based Universal Time Series Forecasting Transformer (Moirai). Trained on our newly introduced Large-scale Open Time Series Archive (LOTSA) featuring over 27B observations across nine domains, Moirai achieves competitive or superior performance as a zero-shot forecaster when compared to full-shot models. Code, data, and model weights can be found at https://github.com/SalesforceAIResearch/uni2ts.

Key facts. Cited by 602 papers in the map (data through 13 Oct 2026). Builds on 50 of its 72 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

50 of 72 references.

References · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  5. 2020Scaling Laws for Neural Language Modelscited by 6,494
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,628
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,999
  8. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,016
  9. 2020GLU Variants Improve Transformercited by 1,759
  10. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,665
  11. 2019Root Mean Square Layer Normalizationcited by 1,449
  12. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,413
38 more
  1. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,206
  2. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,043
  3. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,008
  4. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 918
  5. 2020On Layer Normalization in the Transformer Architecturecited by 910
  6. 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 874
  7. 2019Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecastingcited by 756
  8. 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 751
  9. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 693
  10. 2023A decoder-only foundation model for time-series forecastingcited by 674
  11. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 623
  12. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 497
  13. 2021TS2Vec: Towards Universal Representation of Time Seriescited by 469
  14. 2020A Transformer-based Framework for Multivariate Time Series Representation Learningcited by 450
  15. 2021SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interactioncited by 337
  16. 2023Long-term Forecasting with TiDE: Time-series Dense Encodercited by 330
  17. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 278
  18. 2020Query-Key Normalization for Transformerscited by 270
  19. 2021Monash Time Series Forecasting Archivecited by 261
  20. 2023TimeGPT-1cited by 153
  21. 2022CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecastingcited by 144
  22. 2023UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecastingcited by 141
  23. 2023LargeST: A Benchmark Dataset for Large-Scale Traffic Forecastingcited by 117
  24. 2023SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modelingcited by 114
  25. 2023ForecastPFN: Synthetically-Trained Zero-Shot Forecastingcited by 104
  26. 2023Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospectscited by 98
  27. 2022TableFormer: Robust Transformer Modeling for Table-Text Encodingcited by 91
  28. 2023A Survey on Time-Series Pre-Trained Modelscited by 52
  29. 2023ClimateLearn: Benchmarking Machine Learning for Weather and Climate Modelingcited by 46
  30. 2021Learning Quantile Functions without Quantile Crossing for Distribution-free Time Series Forecastingcited by 32
  31. 2020A machine learning approach for forecasting hierarchical time seriescited by 28
  32. 2023BuildingsBench: A Large-Scale Dataset of 900K Buildings and Benchmark for Short-Term Load Forecastingcited by 26
  33. 2023Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domaincited by 23
  34. 2021SubseasonalClimateUSA: A Dataset for Subseasonal Forecasting and Benchmarkingcited by 21
  35. 2022SDWPF: A Dataset for Spatial Dynamic Wind Power Forecasting Challenge at KDD Cup 2022cited by 21
  36. 2021On the benefits of maximum likelihood estimation for Regression and Forecastingcited by 8
  37. 2023Bridging AI and Energy Forecasting: An Autonomous Workflow with Customized Toolkitcited by 7
  38. 2023Cross-Frequency Time Series Meta-Forecastingcited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 85; 2025: 230; 2026: 287.

602 papers.

Sort:

By area:

  1. 2024Chronos: Learning the Language of Time Seriescited by 737
  2. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 468
  3. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 286
  4. 2024Foundation Models for Time Series Analysis: A Tutorial and Surveycited by 237
  5. 2024GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluationcited by 201
  6. 2024Deep Time Series Models: A Comprehensive Survey and Benchmarkcited by 195
  7. 2024CBraMod: A Criss-Cross Brain Foundation Model for EEG Decodingcited by 195
  8. 2025Sundial: A Family of Highly Capable Time Series Foundation Modelscited by 177
  9. 2025Chronos-2: From Univariate to Universal Forecastingcited by 177
  10. 2024Are Language Models Actually Useful for Time Series Forecasting?cited by 166
  11. 2025TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learningcited by 112
  12. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 107
38 more
  1. 2024Approaching Human-Level Forecasting with Language Modelscited by 103
  2. 2024ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Datacited by 98
  3. 2024Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Expertscited by 92
  4. 2024VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasterscited by 88
  5. 2024Timer-XL: Long-Context Transformers for Unified Time Series Forecastingcited by 84
  6. 2024Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysiscited by 70
  7. 2025Moirai 2.0: When Less Is More for Time Series Forecastingcited by 65
  8. 2025This Time is Different: An Observability Perspective on Time Series Foundation Modelscited by 64
  9. 2025fev-bench: A Realistic Benchmark for Time Series Forecastingcited by 59
  10. 2025A Comprehensive Survey of Deep Learning for Multivariate Time Series Forecasting: A Channel Strategy Perspectivecited by 58
  11. 2025From Tables to Time: Extending TabPFN-v2 to Time Series Forecastingcited by 54
  12. 2025MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answeringcited by 53
  13. 2024TimeSeriesExam: A time series understanding examcited by 48
  14. 2024Toto: Time Series Optimized Transformer for Observabilitycited by 46
  15. 2024SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attentioncited by 44
  16. 2024Retrieval Augmented Time Series Forecastingcited by 44
  17. 2025TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loopcited by 39
  18. 2024Towards a General Time Series Anomaly Detector with Adaptive Bottlenecks and Dual Adversarial Decoderscited by 38
  19. 2025TS-RAG: Retrieval-Augmented Generation based Time Series Foundation Models are Stronger Zero-Shot Forecastercited by 38
  20. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 36
  21. 2024TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Modelcited by 33
  22. 2024In-Context Fine-Tuning for Time-Series Foundation Modelscited by 33
  23. 2024Towards a General Time Series Forecasting Model with Unified Representation and Adaptive Transfercited by 32
  24. 2024Towards Neural Scaling Laws for Time Series Foundation Modelscited by 32
  25. 2024Toward Foundation Model for Multivariate Wearable Sensing of Physiological Signalscited by 30
  26. 2025LightGTS: A Lightweight General Time Series Forecasting Modelcited by 29
  27. 2025Aurora: Towards Universal Generative Multimodal Time Series Forecastingcited by 29
  28. 2024Zero-shot forecasting of chaotic systemscited by 28
  29. 2024TS-Reasoner: Domain-Oriented Time Series Inference Agents for Reasoning and Automated Analysiscited by 28
  30. 2026It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarkscited by 28
  31. 2024Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecastingcited by 26
  32. 2024Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenizationcited by 26
  33. 2025Kronos: A Foundation Model for the Language of Financial Marketscited by 26
  34. 2025TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Modelscited by 26
  35. 2026Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scalingcited by 26
  36. 2024Specialized Foundation Models Struggle to Beat Supervised Baselinescited by 24
  37. 2025Foundation Models for Spatio-Temporal Data Science: A Tutorial and Surveycited by 24
  38. 2024Empowering Time Series Analysis with Foundation Models: A Comprehensive Surveycited by 23

50 most cited of 602 All 602 (JSON)