# Large Pre-trained time series models for cross-domain Time series analysis tasks

arXiv: https://arxiv.org/abs/2311.11413 · HTML: https://mlsanity.com/paper/2311.11413/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 58 references that are in the map.
- First posted on arXiv 19 Nov 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [The UEA multivariate time series classification archive, 2018](https://mlsanity.com/paper/1811.00075/) (2018)
- [Long-term Forecasting with TiDE: Time-series Dense Encoder](https://mlsanity.com/paper/2304.08424/) (2023)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [AutoFormer: Searching Transformers for Visual Recognition](https://mlsanity.com/paper/2107.00651/) (2021)
- [Structured Inference Networks for Nonlinear State Space Models](https://mlsanity.com/paper/1609.09869/) (2016)
- [A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data](https://mlsanity.com/paper/1811.08055/) (2018)
- [A Survey of Vision-Language Pre-Trained Models](https://mlsanity.com/paper/2202.10936/) (2022)
- [Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2206.09113/) (2022)
- [Learning Interpretable Deep State Space Model for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2102.00397/) (2021)

## Built on by (most cited first, 23 of 23)

- [Foundation Models for Time Series Analysis: A Tutorial and Survey](https://mlsanity.com/paper/2403.14735/) (2024)
- [Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysis](https://mlsanity.com/paper/2406.08627/) (2024)
- [LSTPrompt: Large Language Models as Zero-Shot Time Series Forecasters by Long-Short-Term Prompting](https://mlsanity.com/paper/2402.16132/) (2024)
- [Towards a General Time Series Forecasting Model with Unified Representation and Adaptive Transfer](https://mlsanity.com/paper/2405.17478/) (2024)
- [Empowering Time Series Analysis with Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2405.02358/) (2024)
- [Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Models](https://mlsanity.com/paper/2503.11411/) (2025)
- [Evaluating System 1 vs. 2 Reasoning Approaches for Zero-Shot Time Series Forecasting: A Benchmark and Insights](https://mlsanity.com/paper/2503.01895/) (2025)
- [Advancing Time Series Classification with Multimodal Language Modeling](https://mlsanity.com/paper/2403.12371/) (2024)
- [GTM: A General Time-series Model for Enhanced Representation Learning of Time-Series Data](https://mlsanity.com/paper/2502.03264/) (2025)
- [The Multiverse of Time Series Machine Learning: an Archive for Multivariate Time Series Classification](https://mlsanity.com/paper/2603.20352/) (2026)
- [Are Time-Series Foundation Models Deployment-Ready? A Systematic Study of Adversarial Robustness Across Domains](https://mlsanity.com/paper/2505.19397/) (2025)
- [Multi-scale hypergraph meets LLMs: Aligning large language models for time series analysis](https://mlsanity.com/paper/2602.04369/) (2026)
- [Rethinking Post-Training Recipes for Multimodal Time-Series Forecasting](https://mlsanity.com/paper/2605.29401/) (2026)
- [A Time Series Multitask Framework Integrating a Large Language Model, Pre-Trained Time Series Model, and Knowledge Graph](https://mlsanity.com/paper/2503.07682/) (2025)
- [TAT: Temporal-Aligned Transformer for Multi-Horizon Peak Demand Forecasting](https://mlsanity.com/paper/2507.10349/) (2025)
- [SDSC:A Structure-Aware Metric for Semantic Signal Representation Learning](https://mlsanity.com/paper/2507.14516/) (2025)
- [In-context Pre-trained Time-Series Foundation Models adapt to Unseen Tasks](https://mlsanity.com/paper/2602.20307/) (2026)
- [Learning Satellite Pattern-of-Life Identification: A Diffusion-based Approach](https://mlsanity.com/paper/2412.10814/) (2024)
- [BSAT: B-Spline Adaptive Tokenizer for Long-Term Time Series Forecasting](https://mlsanity.com/paper/2601.00698/) (2026)
- [SURGE: An Event-Centric Social Media Sentiment Time Series Benchmark with Interaction Structure](https://mlsanity.com/paper/2605.21198/) (2026)
- [LakeFM: Toward a Foundation Model for Aquatic Ecosystems Using Irregular Multivariate Multi-depth Time Series Data](https://mlsanity.com/paper/2606.11268/) (2026)
- [Is Self-Pretraining really useful to improve diagnosis in medical Time Series?](https://mlsanity.com/paper/2608.06122/) (2026)
- [TimeRSI: Towards Self-Improved Agentic Deployment for Time-Series Foundation Models](https://mlsanity.com/paper/2610.19725/) (2026)

## Abstract (arXiv)

Large pre-trained models have been vital in recent advancements in domains like language and vision, making model training for individual downstream tasks more efficient and provide superior performance. However, tackling time-series analysis tasks usually involves designing and training a separate model from scratch leveraging training data and domain expertise specific to the task. We tackle a significant challenge for pre-training a foundational time-series model from multi-domain time-series datasets: extracting semantically useful tokenized inputs to the model across heterogenous time-series from different domains. We propose Large Pre-trained Time-series Models (LPTM) that introduces a novel method of adaptive segmentation that automatically identifies optimal dataset-specific segmentation strategy during pre-training. This enables LPTM to perform similar to or better than domain-specific state-of-art model when fine-tuned to different downstream time-series analysis tasks and under zero-shot settings. LPTM achieves superior forecasting and time-series classification results taking up to 40% less data and 50% less training time compared to state-of-art baselines. Code: www.github.com/AdityaLab/Samay