# A Survey on Time-Series Pre-Trained Models

arXiv: https://arxiv.org/abs/2305.10716 · HTML: https://mlsanity.com/paper/2305.10716/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 131 of its 221 references that are in the map.
- First posted on arXiv 18 May 2023.

## Builds on (most cited first, 30 of 131)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis](https://mlsanity.com/paper/2210.02186/) (2022)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)

## Built on by (most cited first, 30 of 52)

- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series](https://mlsanity.com/paper/2308.08241/) (2023)
- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [Large Language Models for Time Series: A Survey](https://mlsanity.com/paper/2402.01801/) (2024)
- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](https://mlsanity.com/paper/2310.10196/) (2023)
- [VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2408.17253/) (2024)
- [Empowering Time Series Analysis with Large Language Models: A Survey](https://mlsanity.com/paper/2402.03182/) (2024)
- [Universal Time-Series Representation Learning: A Survey](https://mlsanity.com/paper/2401.03717/) (2024)
- [Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning](https://mlsanity.com/paper/2402.04852/) (2024)
- [Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domain](https://mlsanity.com/paper/2310.05063/) (2023)
- [Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Models](https://mlsanity.com/paper/2402.01749/) (2024)
- [Foundation Models for Time Series: A Survey](https://mlsanity.com/paper/2504.04011/) (2025)
- [FoNE: Precise Single-Token Number Embeddings via Fourier Features](https://mlsanity.com/paper/2502.09741/) (2025)
- [Scaling-laws for Large Time-series Models](https://mlsanity.com/paper/2405.13867/) (2024)
- [Learning Soft Sparse Shapes for Efficient Time-Series Classification](https://mlsanity.com/paper/2505.06892/) (2025)
- [TimeCopilot](https://mlsanity.com/paper/2509.00616/) (2025)
- [HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling with Self-Distillation for Long-Term Forecasting](https://mlsanity.com/paper/2401.05012/) (2024)
- [Right on Time: Revising Time Series Models by Constraining their Explanations](https://mlsanity.com/paper/2402.12921/) (2024)
- [Phase-driven Domain Generalizable Learning for Nonstationary Time Series](https://mlsanity.com/paper/2402.05960/) (2024)
- [UniCL: A Universal Contrastive Learning Framework for Large Time Series Models](https://mlsanity.com/paper/2405.10597/) (2024)
- [Denoising-Aware Contrastive Learning for Noisy Time Series](https://mlsanity.com/paper/2406.04627/) (2024)
- [Guidelines for Augmentation Selection in Contrastive Learning for Time Series Classification](https://mlsanity.com/paper/2407.09336/) (2024)
- [EMIT- Event-Based Masked Auto Encoding for Irregular Time Series](https://mlsanity.com/paper/2409.16554/) (2024)
- [TimelyGPT: Extrapolatable Transformer Pre-training for Long-term Time-Series Forecasting in Healthcare](https://mlsanity.com/paper/2312.00817/) (2023)
- [Towards Foundation Time Series Model: To Synthesize Or Not To Synthesize?](https://mlsanity.com/paper/2403.02534/) (2024)
- [OKG-LLM: Aligning Ocean Knowledge Graph with Observation Data via LLMs for Global Sea Surface Temperature Prediction](https://mlsanity.com/paper/2508.00933/) (2025)
- [Self-Distilled Representation Learning for Time Series](https://mlsanity.com/paper/2311.11335/) (2023)
- [Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptations](https://mlsanity.com/paper/2410.11539/) (2024)
- [Tackling Data Heterogeneity in Federated Time Series Forecasting](https://mlsanity.com/paper/2411.15716/) (2024)

## Abstract (arXiv)

Time-Series Mining (TSM) is an important research area since it shows great potential in practical applications. Deep learning models that rely on massive labeled data have been utilized for TSM successfully. However, constructing a large-scale well-labeled dataset is difficult due to data annotation costs. Recently, pre-trained models have gradually attracted attention in the time series domain due to their remarkable performance in computer vision and natural language processing. In this survey, we provide a comprehensive review of Time-Series Pre-Trained Models (TS-PTMs), aiming to guide the understanding, applying, and studying TS-PTMs. Specifically, we first briefly introduce the typical deep learning models employed in TSM. Then, we give an overview of TS-PTMs according to the pre-training techniques. The main categories we explore include supervised, unsupervised, and self-supervised TS-PTMs. Further, extensive experiments involving 27 methods, 434 datasets, and 679 transfer learning scenarios are conducted to analyze the advantages and disadvantages of transfer learning strategies, Transformer-based models, and representative TS-PTMs. Finally, we point out some potential directions of TS-PTMs for future work.