# Generative Pretrained Hierarchical Transformer for Time Series Forecasting

arXiv: https://arxiv.org/abs/2402.16516 · HTML: https://mlsanity.com/paper/2402.16516/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 55 references that are in the map.
- First posted on arXiv 26 Feb 2024.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction](https://mlsanity.com/paper/2106.09305/) (2021)
- [Monash Time Series Forecasting Archive](https://mlsanity.com/paper/2105.06643/) (2021)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting](https://mlsanity.com/paper/2202.01575/) (2022)
- [Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2206.09113/) (2022)
- [ForecastPFN: Synthetically-Trained Zero-Shot Forecasting](https://mlsanity.com/paper/2311.01933/) (2023)
- [TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders](https://mlsanity.com/paper/2303.00320/) (2023)
- [FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification](https://mlsanity.com/paper/2302.09818/) (2023)
- [Recurrent Neural Networks for Time Series Forecasting](https://mlsanity.com/paper/1901.00069/) (2019)
- [ShapeWordNet: An Interpretable Shapelet Neural Network for Physiological Signal Classification](https://mlsanity.com/paper/2302.05021/) (2023)

## Built on by (most cited first, 15 of 15)

- [ConvTimeNet: A Deep Hierarchical Fully Convolutional Model for Multivariate Time Series Analysis](https://mlsanity.com/paper/2403.01493/) (2024)
- [Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasoner](https://mlsanity.com/paper/2412.20662/) (2024)
- [TableTime: Reformulating Time Series Classification as Training-Free Table Understanding with Large Language Models](https://mlsanity.com/paper/2411.15737/) (2024)
- [SDE: A Simplified and Disentangled Dependency Encoding Framework for State Space Models in Time Series Forecasting](https://mlsanity.com/paper/2408.12068/) (2024)
- [Unveiling the Potential of Text in High-Dimensional Time Series Forecasting](https://mlsanity.com/paper/2501.07048/) (2025)
- [GTM: A General Time-series Model for Enhanced Representation Learning of Time-Series Data](https://mlsanity.com/paper/2502.03264/) (2025)
- [Conditional Denoising Meets Polynomial Modeling: A Flexible Decoupled Framework for Time Series Forecasting](https://mlsanity.com/paper/2410.13253/) (2024)
- [DisenTS: Disentangled Channel Evolving Pattern Modeling for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2410.22981/) (2024)
- [InstructTime++: Time Series Classification with Multimodal Language Modeling via Implicit Feature Enhancement](https://mlsanity.com/paper/2601.14968/) (2026)
- [One Step Closer to Ground Truth: A Multi-Scale Residual-Aware Representation Learning Pipeline for Predicting Time Series Data](https://mlsanity.com/paper/2606.10678/) (2026)
- [A Hybrid Multi-Factor Network with Dynamic Sequence Modeling for Early Warning of Intraoperative Hypotension](https://mlsanity.com/paper/2409.11064/) (2024)
- [PhyxMamba: Chaotic System Reconstruction from Short Context Observations with Generative State-Space Models](https://mlsanity.com/paper/2505.23863/) (2025)
- [QKCV Attention: Enhancing Time Series Forecasting with Static Categorical Embeddings for Both Lightweight and Pre-trained Foundation Models](https://mlsanity.com/paper/2510.20222/) (2025)
- [DCD: Decomposition-based Causal Discovery from Autocorrelated and Non-Stationary Temporal Data](https://mlsanity.com/paper/2602.01433/) (2026)
- [Learning to Forecast Crop Growth from Earth Observation Data](https://mlsanity.com/paper/2608.14281/) (2026)

## Abstract (arXiv)

Recent efforts have been dedicated to enhancing time series forecasting accuracy by introducing advanced network architectures and self-supervised pretraining strategies. Nevertheless, existing approaches still exhibit two critical drawbacks. Firstly, these methods often rely on a single dataset for training, limiting the model's generalizability due to the restricted scale of the training data. Secondly, the one-step generation schema is widely followed, which necessitates a customized forecasting head and overlooks the temporal dependencies in the output series, and also leads to increased training costs under different horizon length settings. To address these issues, we propose a novel generative pretrained hierarchical transformer architecture for forecasting, named GPHT. There are two aspects of key designs in GPHT. On the one hand, we advocate for constructing a mixed dataset under the channel-independent assumption for pretraining our model, comprising various datasets from diverse data scenarios. This approach significantly expands the scale of training data, allowing our model to uncover commonalities in time series data and facilitating improved transfer to specific datasets. On the other hand, GPHT employs an auto-regressive forecasting approach, effectively modeling temporal dependencies in the output series. Importantly, no customized forecasting head is required, enabling textita single model to forecast at arbitrary horizon settings. We conduct sufficient experiments on eight datasets with mainstream self-supervised pretraining models and supervised models. The results demonstrated that GPHT surpasses the baseline models across various fine-tuning and zero/few-shot learning settings in the traditional long-term forecasting task. We make our codes publicly availablefootnotehttps://github.com/icantnamemyself/GPHT.