# AutoHFormer: Efficient Hierarchical Autoregressive Transformer for Time Series Prediction

arXiv: https://arxiv.org/abs/2506.16001 · HTML: https://mlsanity.com/paper/2506.16001/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 50 references that are in the map.
- First posted on arXiv 19 Jun 2025.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis](https://mlsanity.com/paper/2210.02186/) (2022)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [Time Series Forecasting With Deep Learning: A Survey](https://mlsanity.com/paper/2004.13408/) (2020)
- [TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting](https://mlsanity.com/paper/2405.14616/) (2024)
- [A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection](https://mlsanity.com/paper/2307.03759/) (2023)
- [Rademacher Complexity for Adversarially Robust Generalization](https://mlsanity.com/paper/1810.11914/) (2018)
- [Is Mamba Effective for Time Series Forecasting?](https://mlsanity.com/paper/2403.11144/) (2024)
- [SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series](https://mlsanity.com/paper/2403.15360/) (2024)
- [TimeFilter: Patch-Specific Spatial-Temporal Graph Filtration for Time Series Forecasting](https://mlsanity.com/paper/2501.13041/) (2025)
- [Self-supervised Trajectory Representation Learning with Temporal Regularities and Travel Semantics](https://mlsanity.com/paper/2211.09510/) (2022)
- [SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention](https://mlsanity.com/paper/2402.10198/) (2024)
- [Graph Attention Recurrent Neural Networks for Correlated Time Series Forecasting -- Full version](https://mlsanity.com/paper/2103.10760/) (2021)
- [A Survey of Generative Techniques for Spatial-Temporal Data Mining](https://mlsanity.com/paper/2405.09592/) (2024)
- [A Survey on Point-of-Interest Recommendation: Models, Architectures, and Security](https://mlsanity.com/paper/2410.02191/) (2024)
- [Efficient Traffic Prediction Through Spatio-Temporal Distillation](https://mlsanity.com/paper/2501.10459/) (2025)
- [Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mapping](https://mlsanity.com/paper/2305.10721/) (2023)
- [HMamba: Hyperbolic Mamba for Sequential Recommendation](https://mlsanity.com/paper/2505.09205/) (2025)
- [M2Rec: Multi-scale Mamba for Efficient Sequential Recommendation](https://mlsanity.com/paper/2505.04445/) (2025)

## Built on by (most cited first, 8 of 8)

- [ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions](https://mlsanity.com/paper/2511.14342/) (2025)
- [Universal Redundancies in Time Series Foundation Models](https://mlsanity.com/paper/2602.01605/) (2026)
- [FLDmamba: Integrating Fourier and Laplace Transform Decomposition with Mamba for Enhanced Time Series Prediction](https://mlsanity.com/paper/2507.12803/) (2025)
- [MODE: Efficient Time Series Prediction with Mamba Enhanced by Low-Rank Neural ODEs](https://mlsanity.com/paper/2601.00920/) (2026)
- [Neural CDEs as Correctors for Learned Time Series Models](https://mlsanity.com/paper/2512.12116/) (2025)
- [UniMamba: A Unified Spatial-Temporal Modeling Framework with State-Space and Attention Integration](https://mlsanity.com/paper/2604.16325/) (2026)
- [A Hybrid Attention Model Learning Unified Time-aware Patch Representation for Irregular Multivariate Time Series Forecasting](https://mlsanity.com/paper/2609.22836/) (2026)
- [CTIA-ODE: Continuous-Time Inter-Variable Attention for Long-Term Time Series Forecasting](https://mlsanity.com/paper/2610.19831/) (2026)

## Abstract (arXiv)

Time series forecasting requires architectures that simultaneously achieve three competing objectives: (1) strict temporal causality for reliable predictions, (2) sub-quadratic complexity for practical scalability, and (3) multi-scale pattern recognition for accurate long-horizon forecasting. We introduce AutoHFormer, a hierarchical autoregressive transformer that addresses these challenges through three key innovations: 1) Hierarchical Temporal Modeling: Our architecture decomposes predictions into segment-level blocks processed in parallel, followed by intra-segment sequential refinement. This dual-scale approach maintains temporal coherence while enabling efficient computation. 2) Dynamic Windowed Attention: The attention mechanism employs learnable causal windows with exponential decay, reducing complexity while preserving precise temporal relationships. This design avoids both the anti-causal violations of standard transformers and the sequential bottlenecks of RNN hybrids. 3) Adaptive Temporal Encoding: a novel position encoding system is adopted to capture time patterns at multiple scales. It combines fixed oscillating patterns for short-term variations with learnable decay rates for long-term trends. Comprehensive experiments demonstrate that AutoHFormer 10.76X faster training and 6.06X memory reduction compared to PatchTST on PEMS08, while maintaining consistent accuracy across 96-720 step horizons in most of cases. These breakthroughs establish new benchmarks for efficient and precise time series modeling. Implementations of our method and all baselines in hierarchical autoregressive mechanism are available at https://github.com/lizzyhku/Autotime.