# FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification

arXiv: https://arxiv.org/abs/2302.09818 · HTML: https://mlsanity.com/paper/2302.09818/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 53 references that are in the map.
- First posted on arXiv 20 Feb 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks](https://mlsanity.com/paper/2005.11650/) (2020)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [InceptionTime: Finding AlexNet for Time Series Classification](https://mlsanity.com/paper/1909.04939/) (2019)
- [ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels](https://mlsanity.com/paper/1910.13051/) (2019)
- [The UEA multivariate time series classification archive, 2018](https://mlsanity.com/paper/1811.00075/) (2018)
- [Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019](https://mlsanity.com/paper/1911.13288/) (2019)
- [ReZero is All You Need: Fast Convergence at Large Depth](https://mlsanity.com/paper/2003.04887/) (2020)
- [MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classification](https://mlsanity.com/paper/2012.08791/) (2020)
- [How Much Position Information Do Convolutional Neural Networks Encode?](https://mlsanity.com/paper/2001.08248/) (2020)
- [Benchmarking Multivariate Time Series Classification Algorithms](https://mlsanity.com/paper/2007.13156/) (2020)
- [Multivariate LSTM-FCNs for Time Series Classification](https://mlsanity.com/paper/1801.04503/) (2018)
- [HIVE-COTE 2.0: a new meta ensemble for time series classification](https://mlsanity.com/paper/2104.07551/) (2021)
- [Multi-Scale Convolutional Neural Networks for Time Series Classification](https://mlsanity.com/paper/1603.06995/) (2016)
- [Self-supervised ECG Representation Learning for Emotion Recognition](https://mlsanity.com/paper/2002.03898/) (2020)
- [MultiRocket: Multiple pooling operators and transformations for fast and effective time series classification](https://mlsanity.com/paper/2102.00457/) (2021)
- [Self-attention for raw optical Satellite Time Series Classification](https://mlsanity.com/paper/1910.10536/) (2019)
- [Explaining Time Series Predictions with Dynamic Masks](https://mlsanity.com/paper/2106.05303/) (2021)
- [Flowformer: Linearizing Transformers with Conservation Flows](https://mlsanity.com/paper/2202.06258/) (2022)
- [Gated Transformer Networks for Multivariate Time Series Classification](https://mlsanity.com/paper/2103.14438/) (2021)
- [Multivariate Time Series Classification with WEASEL+MUSE](https://mlsanity.com/paper/1711.11343/) (2017)

## Built on by (most cited first, 21 of 21)

- [TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders](https://mlsanity.com/paper/2303.00320/) (2023)
- [Universal Time-Series Representation Learning: A Survey](https://mlsanity.com/paper/2401.03717/) (2024)
- [ConvTimeNet: A Deep Hierarchical Fully Convolutional Model for Multivariate Time Series Analysis](https://mlsanity.com/paper/2403.01493/) (2024)
- [Generative Pretrained Hierarchical Transformer for Time Series Forecasting](https://mlsanity.com/paper/2402.16516/) (2024)
- [TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation](https://mlsanity.com/paper/2410.05711/) (2024)
- [A Multi-Scale Decomposition MLP-Mixer for Time Series Analysis](https://mlsanity.com/paper/2310.11959/) (2023)
- [Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasoner](https://mlsanity.com/paper/2412.20662/) (2024)
- [TableTime: Reformulating Time Series Classification as Training-Free Table Understanding with Large Language Models](https://mlsanity.com/paper/2411.15737/) (2024)
- [Hierarchical Multimodal LLMs with Semantic Space Alignment for Enhanced Time Series Classification](https://mlsanity.com/paper/2410.18686/) (2024)
- [Revisiting Backdoor Attacks on Time Series Classification in the Frequency Domain](https://mlsanity.com/paper/2503.09712/) (2025)
- [DisMS-TS: Eliminating Redundant Multi-Scale Features for Time Series Classification](https://mlsanity.com/paper/2507.04600/) (2025)
- [The Multiverse of Time Series Machine Learning: an Archive for Multivariate Time Series Classification](https://mlsanity.com/paper/2603.20352/) (2026)
- [TraffiDent: A Dataset for Understanding the Interplay Between Traffic Dynamics and Incidents](https://mlsanity.com/paper/2407.11477/) (2024)
- [VSFormer: Value and Shape-Aware Transformer with Prior-Enhanced Self-Attention for Multivariate Time Series Classification](https://mlsanity.com/paper/2412.16515/) (2024)
- [TimeFlow: Towards Stochastic-Aware and Efficient Time Series Generation via Flow Matching Modeling](https://mlsanity.com/paper/2511.07968/) (2025)
- [InstructTime++: Time Series Classification with Multimodal Language Modeling via Implicit Feature Enhancement](https://mlsanity.com/paper/2601.14968/) (2026)
- [ST-Tree with Interpretability for Multivariate Time Series Classification](https://mlsanity.com/paper/2411.11620/) (2024)
- [Causal and Local Correlations Based Network for Multivariate Time Series Classification](https://mlsanity.com/paper/2411.18008/) (2024)
- [Learning Compositional Transferability of Time Series for Source-Free Domain Adaptation](https://mlsanity.com/paper/2504.14994/) (2025)
- [Multimodal Forecasting of Sparse Intraoperative Hypotension Events Powered by Language Model](https://mlsanity.com/paper/2505.22116/) (2025)
- [Frequency-Domain Multi-Modality Transportation Modeling](https://mlsanity.com/paper/2607.08475/) (2026)

## Abstract (arXiv)

Deep learning-based algorithms, e.g., convolutional networks, have significantly facilitated multivariate time series classification (MTSC) task. Nevertheless, they suffer from the limitation in modeling long-range dependence due to the nature of convolution operations. Recent advancements have shown the potential of transformers to capture long-range dependence. However, it would incur severe issues, such as fixed scale representations, temporal-invariant and quadratic time complexity, with transformers directly applicable to the MTSC task because of the distinct properties of time series data. To tackle these issues, we propose FormerTime, an hierarchical representation model for improving the classification capacity for the MTSC task. In the proposed FormerTime, we employ a hierarchical network architecture to perform multi-scale feature maps. Besides, a novel transformer encoder is further designed, in which an efficient temporal reduction attention layer and a well-informed contextual positional encoding generating strategy are developed. To sum up, FormerTime exhibits three aspects of merits: (1) learning hierarchical multi-scale representations from time series data, (2) inheriting the strength of both transformers and convolutional networks, and (3) tacking the efficiency challenges incurred by the self-attention mechanism. Extensive experiments performed on 10 publicly available datasets from UEA archive verify the superiorities of the FormerTime compared to previous competitive baselines.