mlsanityData to 14 Oct 2026

Timer: Generative Pre-trained Transformers Are Large Time Series Models

arXiv 2402.02368 ↗First posted 4 Feb 2024cs.LG stat.MLcited by 208

Not on the map. Why?

Abstract (arXiv)

Deep learning has contributed remarkably to the advancement of time series analysis. Still, deep models can encounter performance bottlenecks in real-world data-scarce scenarios, which can be concealed due to the performance saturation with small models on current benchmarks. Meanwhile, large models have demonstrated great powers in these scenarios through large-scale pre-training. Continuous progress has been achieved with the emergence of large language models, exhibiting unprecedented abilities such as few-shot generalization, scalability, and task generality, which are however absent in small deep models. To change the status quo of training scenario-specific small models from scratch, this paper aims at the early development of large time series models (LTSM). During pre-training, we curate large-scale datasets with up to 1 billion time points, unify heterogeneous time series into single-series sequence (S3) format, and develop the GPT-style architecture toward LTSMs. To meet diverse application needs, we convert forecasting, imputation, and anomaly detection of time series into a unified generative task. The outcome of this study is a Time Series Transformer (Timer), which is generative pre-trained by next token prediction and adapted to various downstream tasks with promising capabilities as an LTSM. Code and datasets are available at: https://github.com/thuml/Large-Time-Series-Model.

Key facts. Cited by 208 papers in the map (data through 14 Oct 2026). Builds on 44 of its 71 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

44 of 71 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2023A Survey of Large Language Modelscited by 2,576
  10. 2022Emergent Abilities of Large Language Modelscited by 2,512
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  12. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
32 more
  1. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
  2. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,476
  3. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  4. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,058
  5. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 931
  6. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
  7. 2023A decoder-only foundation model for time-series forecastingcited by 692
  8. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  9. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 632
  10. 2021TS2Vec: Towards Universal Representation of Time Seriescited by 471
  11. 2020A Transformer-based Framework for Multivariate Time Series Representation Learningcited by 453
  12. 2018The UCR Time Series Archivecited by 431
  13. 2021Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancycited by 394
  14. 2021SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interactioncited by 338
  15. 2023Long-term Forecasting with TiDE: Time-series Dense Encodercited by 332
  16. 2021Monash Time Series Forecasting Archivecited by 267
  17. 2022Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistencycited by 250
  18. 2023TimeGPT-1cited by 158
  19. 2020Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progresscited by 151
  20. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 150
  21. 2022CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecastingcited by 144
  22. 2023LargeST: A Benchmark Dataset for Large-Scale Traffic Forecastingcited by 118
  23. 2023SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modelingcited by 115
  24. 2023ForecastPFN: Synthetically-Trained Zero-Shot Forecastingcited by 104
  25. 2023Contrast Everything: A Hierarchical Contrastive Framework for Medical Time-Seriescited by 59
  26. 2023ClimateLearn: Benchmarking Machine Learning for Weather and Climate Modelingcited by 47
  27. 2020A machine learning approach for forecasting hierarchical time seriescited by 29
  28. 2023BuildingsBench: A Large-Scale Dataset of 900K Buildings and Benchmark for Short-Term Load Forecastingcited by 26
  29. 2023Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domaincited by 23
  30. 2021SubseasonalClimateUSA: A Dataset for Subseasonal Forecasting and Benchmarkingcited by 21
  31. 2022SDWPF: A Dataset for Spatial Dynamic Wind Power Forecasting Challenge at KDD Cup 2022cited by 21
  32. 2023LibCity: A Unified Library Towards Efficient and Comprehensive Urban Spatial-Temporal Predictioncited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 30; 2025: 82; 2026: 96.

208 papers.

Sort:

By area:

  1. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 482
  2. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 287
  3. 2025Sundial: A Family of Highly Capable Time Series Foundation Modelscited by 180
  4. 2024Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Expertscited by 95
  5. 2024VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasterscited by 88
  6. 2024Timer-XL: Long-Context Transformers for Unified Time Series Forecastingcited by 87
  7. 2025Time-MQA: Time Series Multi-Task Question Answering with Context Enhancementcited by 82
  8. 2024Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysiscited by 74
  9. 2025This Time is Different: An Observability Perspective on Time Series Foundation Modelscited by 68
  10. 2025A Comprehensive Survey of Deep Learning for Multivariate Time Series Forecasting: A Channel Strategy Perspectivecited by 58
  11. 2024Deep Learning for Multivariate Time Series Imputation: A Surveycited by 57
  12. 2024TimeSeriesExam: A time series understanding examcited by 49
38 more
  1. 2025DBLoss: Decomposition-based Loss Function for Time Series Forecastingcited by 45
  2. 2025TS-RAG: Retrieval-Augmented Generation based Time Series Foundation Models are Stronger Zero-Shot Forecastercited by 40
  3. 2025ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Datasetcited by 37
  4. 2024Towards Neural Scaling Laws for Time Series Foundation Modelscited by 33
  5. 2025Aurora: Towards Universal Generative Multimodal Time Series Forecastingcited by 29
  6. 2026Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scalingcited by 28
  7. 2024Learning Pattern-Specific Experts for Time Series Forecasting Under Patch-level Distribution Shiftcited by 24
  8. 2024Empowering Time Series Analysis with Foundation Models: A Comprehensive Surveycited by 23
  9. 2025Foundation Models for Time Series: A Surveycited by 23
  10. 2025MIRA: Medical Time Series Foundation Model for Real-World Health Datacited by 22
  11. 2024Exploring Representations and Interventions in Time Series Foundation Modelscited by 20
  12. 2025Deep Learning and Foundation Models for Weather Prediction: A Surveycited by 16
  13. 2025Out-of-Distribution Generalization in Time Series: A Surveycited by 16
  14. 2025BLAST: Balanced Sampling Time Series Corpus for Universal Forecasting Modelscited by 16
  15. 2025Rethinking Evaluation in the Era of Time Series Foundation Models: (Un)known Information Leakage Challengescited by 16
  16. 2024TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representationcited by 14
  17. 2024Federated Foundation Models on Heterogeneous Time Seriescited by 13
  18. 2025CauKer: Classification Time Series Foundation Models Can Be Pretrained on Synthetic Datacited by 13
  19. 2025Adaptive Root Cause Localization for Microservice Systems with Multi-Agent Recursion-of-Thoughtcited by 13
  20. 2026Reverso: Efficient Time Series Foundation Models for Zero-shot Forecastingcited by 13
  21. 2025VisionTS++: Cross-Modal Time Series Foundation Model with Continual Pre-trained Vision Backbonescited by 12
  22. 2025TS-Reasoner: Aligning Time Series Foundation Models with LLM Reasoningcited by 12
  23. 2025TimeHF: Billion-Scale Time Series Models Guided by Human Feedbackcited by 10
  24. 2026QuitoBench: A High-Quality Open Time Series Forecasting Benchmarkcited by 10
  25. 2026Falcon-X: A Time Series Foundation Model for Heterogeneous Multivariate Modelingcited by 10
  26. 2024A Mamba Foundation Model for Time Series Forecastingcited by 9
  27. 2025CMoS: Rethinking Time Series Prediction Through the Lens of Chunk-wise Spatial Correlationscited by 9
  28. 2025TimeCopilotcited by 9
  29. 2026Learning the Context of Errors: Black-Box Online Adaptation of Time Series Foundation Modelscited by 9
  30. 2025There are no Champions in Supervised Long-Term Time Series Forecastingcited by 8
  31. 2025Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Datacited by 8
  32. 2025Selective Learning for Deep Time Series Forecastingcited by 8
  33. 2025Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectivescited by 7
  34. 2025STAR: Boosting Time Series Foundation Models for Anomaly Detection through State-aware Adaptercited by 7
  35. 2025SEMPO: Lightweight Foundation Models for Time Series Forecastingcited by 7
  36. 2024A Survey of Financial AI: Architectures, Advances and Open Challengescited by 6
  37. 2025Unlocking Pretrained Vision Transformers for Time Series Classificationcited by 6
  38. 2025Foundation models for time series forecasting: Application in conformal predictioncited by 6

50 most cited of 208 All 208 (JSON)