mlsanityData to 14 Oct 2026

A Survey on Time-Series Pre-Trained Models

arXiv 2305.10716 ↗First posted 18 May 2023cs.LG cs.AIcited by 52

Not on the map. Why?

Abstract (arXiv)

Time-Series Mining (TSM) is an important research area since it shows great potential in practical applications. Deep learning models that rely on massive labeled data have been utilized for TSM successfully. However, constructing a large-scale well-labeled dataset is difficult due to data annotation costs. Recently, pre-trained models have gradually attracted attention in the time series domain due to their remarkable performance in computer vision and natural language processing. In this survey, we provide a comprehensive review of Time-Series Pre-Trained Models (TS-PTMs), aiming to guide the understanding, applying, and studying TS-PTMs. Specifically, we first briefly introduce the typical deep learning models employed in TSM. Then, we give an overview of TS-PTMs according to the pre-training techniques. The main categories we explore include supervised, unsupervised, and self-supervised TS-PTMs. Further, extensive experiments involving 27 methods, 434 datasets, and 679 transfer learning scenarios are conducted to analyze the advantages and disadvantages of transfer learning strategies, Transformer-based models, and representative TS-PTMs. Finally, we point out some potential directions of TS-PTMs for future work.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 131 of its 221 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

131 of 221 references.

References · 131

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  8. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  10. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  11. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  12. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
38 more
  1. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  2. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  3. 2016Matching Networks for One Shot Learningcited by 2,255
  4. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  5. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  6. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
  7. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  8. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,476
  9. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  10. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,210
  11. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  12. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  13. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  14. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,058
  15. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  16. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 692
  17. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 632
  18. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 607
  19. 2014Deep Domain Confusion: Maximizing for Domain Invariancecited by 600
  20. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 582
  21. 2018Deep learning for time series classification: a reviewcited by 576
  22. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  23. 2016Recurrent Neural Networks for Multivariate Time Series with Missing Valuescited by 554
  24. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  25. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
  26. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  27. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  28. 2022Transformers in Time Series: A Surveycited by 497
  29. 2021TS2Vec: Towards Universal Representation of Time Seriescited by 471
  30. 2020Time Series Forecasting With Deep Learning: A Surveycited by 459
  31. 2020A Transformer-based Framework for Multivariate Time Series Representation Learningcited by 453
  32. 2018The UCR Time Series Archivecited by 431
  33. 2021Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancycited by 394
  34. 2019InceptionTime: Finding AlexNet for Time Series Classificationcited by 387
  35. 2015Recent Advances in Convolutional Neural Networkscited by 360
  36. 2016Time Series Classification from Scratch with Deep Neural Networks: A Strong Baselinecited by 360
  37. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 347
  38. 2021Time-Series Representation Learning via Temporal and Contextual Contrastingcited by 345

50 most cited of 131 All 131 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 22; 2025: 13; 2026: 9.

52 papers.

Sort:

By area:

  1. 2023A decoder-only foundation model for time-series forecastingcited by 692
  2. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 614
  3. 2023TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Seriescited by 169
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  5. 2024Large Language Models for Time Series: A Surveycited by 114
  6. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 110
  7. 2024VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasterscited by 88
  8. 2024Empowering Time Series Analysis with Large Language Models: A Surveycited by 60
  9. 2024Universal Time-Series Representation Learning: A Surveycited by 30
  10. 2024Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learningcited by 25
  11. 2023Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domaincited by 23
  12. 2024Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Modelscited by 23
38 more
  1. 2025Foundation Models for Time Series: A Surveycited by 23
  2. 2025FoNE: Precise Single-Token Number Embeddings via Fourier Featurescited by 22
  3. 2024Scaling-laws for Large Time-series Modelscited by 18
  4. 2025Learning Soft Sparse Shapes for Efficient Time-Series Classificationcited by 10
  5. 2025TimeCopilotcited by 9
  6. 2024HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling with Self-Distillation for Long-Term Forecastingcited by 8
  7. 2024Right on Time: Revising Time Series Models by Constraining their Explanationscited by 7
  8. 2024Phase-driven Domain Generalizable Learning for Nonstationary Time Seriescited by 5
  9. 2024UniCL: A Universal Contrastive Learning Framework for Large Time Series Modelscited by 5
  10. 2024Denoising-Aware Contrastive Learning for Noisy Time Seriescited by 5
  11. 2024Guidelines for Augmentation Selection in Contrastive Learning for Time Series Classificationcited by 5
  12. 2024EMIT- Event-Based Masked Auto Encoding for Irregular Time Seriescited by 5
  13. 2023TimelyGPT: Extrapolatable Transformer Pre-training for Long-term Time-Series Forecasting in Healthcarecited by 4
  14. 2024Towards Foundation Time Series Model: To Synthesize Or Not To Synthesize?cited by 4
  15. 2025OKG-LLM: Aligning Ocean Knowledge Graph with Observation Data via LLMs for Global Sea Surface Temperature Predictioncited by 4
  16. 2023Self-Distilled Representation Learning for Time Seriescited by 3
  17. 2024Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptationscited by 3
  18. 2024Tackling Data Heterogeneity in Federated Time Series Forecastingcited by 3
  19. 2025One-Embedding-Fits-All: Efficient Zero-Shot Time Series Forecasting by a Model Zoocited by 3
  20. 2024United We Pretrain, Divided We Fail! Representation Learning for Time Series by Pretraining on 75 Datasets at Oncecited by 2
  21. 2025Agentic Satellite-Augmented Low-Altitude Economy and Terrestrial Networks: A Survey on Generative Approachescited by 2
  22. 2024On the Intersection of Signal Processing and Machine Learning: A Use Case-Driven Analysis Approachcited by 1
  23. 2025CoIFNet: A Unified Framework for Multivariate Time Series Forecasting with Missing Valuescited by 1
  24. 2025Re-Initialization Token Learning for Tool-Augmented Large Language Modelscited by 1
  25. 2025SDSC:A Structure-Aware Metric for Semantic Signal Representation Learningcited by 1
  26. 2025Time Series Foundation Models for Process Model Forecastingcited by 1
  27. 2023Temporal Treasure Hunt: Content-based Time Series Retrieval System for Discovering Insightscited by 0
  28. 2023Ego-Network Transformer for Subsequence Classification in Time Series Datacited by 0
  29. 2024TripCast: Pre-training of Masked 2D Transformers for Trip Time Series Forecastingcited by 0
  30. 2025Collaborative Imputation of Urban Time Series through Cross-city Meta-learningcited by 0
  31. 2025Multimodal Forecasting of Sparse Intraoperative Hypotension Events Powered by Language Modelcited by 0
  32. 2026Generalizing Dynamics Modeling More Easily from Representation Perspectivecited by 0
  33. 2026Natural Language Interfaces for Spatial and Temporal Databases: A Comprehensive Overview of Methods, Taxonomy, and Future Directionscited by 0
  34. 2026EvoTSC: Evolving Feature Learning Models for Time Series Classification via Genetic Programmingcited by 0
  35. 2026Prototype-Guided Classification Sub-Task Decoupling Framework: Enhancing Generalization and Interpretability for Multivariate Time Seriescited by 0
  36. 2026A Simple State Space Model Excels at Multivariate Time Series Classificationcited by 0
  37. 2026Evaluating the Generalizability of Foundation Models for Extreme Environmental Events: Case Study of California Wildfire PM2.5cited by 0
  38. 2026Is Self-Pretraining really useful to improve diagnosis in medical Time Series?cited by 0

50 most cited of 52 All 52 (JSON)