mlsanityData to 14 Oct 2026

TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting

arXiv 2310.04948 ↗First posted 8 Oct 2023cs.LG cs.CLcited by 174

Not on the map. Why?

Abstract (arXiv)

The past decade has witnessed significant advances in time series modeling with deep learning. While achieving state-of-the-art results, the best-performing architectures vary highly across applications and domains. Meanwhile, for natural language processing, the Generative Pre-trained Transformer (GPT) has demonstrated impressive performance via training one general-purpose model across various textual datasets. It is intriguing to explore whether GPT-type architectures can be effective for time series, capturing the intrinsic dynamic attributes and leading to significant accuracy improvements. In this paper, we propose a novel framework, TEMPO, that can effectively learn time series representations. We focus on utilizing two essential inductive biases of the time series task for pre-trained models: (i) decomposition of the complex interaction between trend, seasonal and residual components; and (ii) introducing the design of prompts to facilitate distribution adaptation in different types of time series. TEMPO expands the capability for dynamically modeling real-world temporal phenomena from data within diverse domains. Our experiments demonstrate the superior performance of TEMPO over state-of-the-art methods on zero shot setting for a number of time series benchmark datasets. This performance gain is observed not only in scenarios involving previously unseen datasets but also in scenarios with multi-modal inputs. This compelling finding highlights TEMPO's potential to constitute a foundational model-building framework.

Key facts. Cited by 174 papers in the map (data through 14 Oct 2026). Builds on 42 of its 73 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

42 of 73 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  12. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
30 more
  1. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
  2. 2020Reformer: The Efficient Transformercited by 1,668
  3. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,476
  4. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  5. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,210
  6. 2021Grounded Language-Image Pre-trainingcited by 1,156
  7. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,058
  8. 2021Learning to Prompt for Continual Learningcited by 944
  9. 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 753
  10. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
  11. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  12. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 632
  13. 2022Non-stationary Transformers: Exploring the Stationarity in Time Series Forecastingcited by 391
  14. 2021SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interactioncited by 338
  15. 2023Long-term Forecasting with TiDE: Time-series Dense Encodercited by 332
  16. 2021Spectral Temporal Graph Neural Network for Multivariate Time-series Forecastingcited by 299
  17. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  18. 2022Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistencycited by 250
  19. 2022PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecastingcited by 172
  20. 2023TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Seriescited by 169
  21. 2019Causal Discovery from Heterogeneous/Nonstationary Data with Independent Changescited by 166
  22. 2022Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structurescited by 160
  23. 2023TimeGPT-1cited by 158
  24. 2022ETSformer: Exponential Smoothing Transformers for Time-series Forecastingcited by 148
  25. 2023TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecastingcited by 139
  26. 2023Dish-TS: A General Paradigm for Alleviating Distribution Shift in Time Series Forecastingcited by 93
  27. 2023Temporal Data Meets LLM -- Explainable Financial Time Series Forecastingcited by 77
  28. 2023Frozen Language Model Helps ECG Zero-Shot Learningcited by 62
  29. 2022DEPTS: Deep Expansion Learning for Periodic Time Series Forecastingcited by 29
  30. 2022Sparse Interaction Additive Networks via Feature Interaction Detection and Sparse Selectioncited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 60; 2025: 78; 2026: 33.

174 papers.

Sort:

By area:

  1. 2024Deep Time Series Models: A Comprehensive Survey and Benchmarkcited by 197
  2. 2024Are Language Models Actually Useful for Time Series Forecasting?cited by 168
  3. 2024Large Language Models for Time Series: A Surveycited by 114
  4. 2023Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecastingcited by 113
  5. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 110
  6. 2024CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuningcited by 108
  7. 2024CycleNet: Enhancing Time Series Forecasting through Modeling Periodic Patternscited by 106
  8. 2024From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflectioncited by 98
  9. 2024Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Expertscited by 95
  10. 2024UniTS: A Unified Multi-Task Time Series Modelcited by 94
  11. 2024Language Models Still Struggle to Zero-shot Reason about Time Seriescited by 88
  12. 2024Spatial-Temporal Large Language Model for Traffic Predictioncited by 83
38 more
  1. 2024Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysiscited by 74
  2. 2024AutoTimes: Autoregressive Time Series Forecasters via Large Language Modelscited by 65
  3. 2024Empowering Time Series Analysis with Large Language Models: A Surveycited by 60
  4. 2025A Comprehensive Survey of Deep Learning for Multivariate Time Series Forecasting: A Channel Strategy Perspectivecited by 58
  5. 2025MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answeringcited by 55
  6. 2024Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Reviewcited by 53
  7. 2024TimeSeriesExam: A time series understanding examcited by 49
  8. 2024Time Series Forecasting with LLMs: Understanding and Enhancing Model Capabilitiescited by 48
  9. 2025Language in the Flow of Time: Time-Series-Paired Texts Weaved into a Unified Temporal Narrativecited by 40
  10. 2025Context-Alignment: Activating and Enhancing LLM Capabilities in Time Seriescited by 39
  11. 2024Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlookcited by 33
  12. 2024TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Modelcited by 33
  13. 2024Rethinking Channel Dependence for Multivariate Time Series Forecasting: Learning from Leading Indicatorscited by 32
  14. 2024TS-Reasoner: Domain-Oriented Time Series Inference Agents for Reasoning and Automated Analysiscited by 29
  15. 2024Large Language Model Guided Knowledge Distillation for Time Series Anomaly Detectioncited by 26
  16. 2024Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learningcited by 25
  17. 2024Specialized Foundation Models Struggle to Beat Supervised Baselinescited by 24
  18. 2024Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Modelscited by 23
  19. 2024TimeSeriesBench: An Industrial-Grade Benchmark for Time Series Anomaly Detection Modelscited by 23
  20. 2024Empowering Time Series Analysis with Foundation Models: A Comprehensive Surveycited by 23
  21. 2024Can LLMs Serve As Time Series Anomaly Detectors?cited by 22
  22. 2024Mobility-LLM: Learning Visiting Intentions and Travel Preferences from Human Mobility Data with Large Language Modelscited by 22
  23. 2025FoNE: Precise Single-Token Number Embeddings via Fourier Featurescited by 22
  24. 2024Towards Modeling Learner Performance with Large Language Modelscited by 21
  25. 2025A Survey of Reasoning and Agentic Systems in Time Series with Large Language Modelscited by 21
  26. 2025Efficient Multivariate Time Series Forecasting via Calibrated Language Models with Privileged Knowledge Distillationcited by 20
  27. 2024xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memoriescited by 19
  28. 2025BRIDGE: Bootstrapping Text to Control Time-Series Generation via Multi-Agent Iterative Optimization and Diffusion Modelingcited by 19
  29. 2025A Survey of AIOps in the Era of Large Language Modelscited by 19
  30. 2024S2IP-LLM: Semantic Space Informed Prompt Learning with LLM for Time Series Forecastingcited by 17
  31. 2025Achieving Time Series Reasoning Requires Rethinking Model Design, Tasks Formulation, and Evaluationcited by 17
  32. 2024Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Modelscited by 15
  33. 2025GPT-PPG: A GPT-based Foundation Model for Photoplethysmography Signalscited by 15
  34. 2024A Survey of Generative Techniques for Spatial-Temporal Data Miningcited by 14
  35. 2025Time Series Generation Under Data Scarcity: A Unified Generative Modeling Approachcited by 14
  36. 2024Federated Foundation Models on Heterogeneous Time Seriescited by 13
  37. 2025CauKer: Classification Time Series Foundation Models Can Be Pretrained on Synthetic Datacited by 13
  38. 2025From Values to Tokens: An LLM-Driven Framework for Context-aware Time Series Forecasting via Symbolic Discretizationcited by 13

50 most cited of 174 All 174 (JSON)