# Gated Transformer Networks for Multivariate Time Series Classification

arXiv: https://arxiv.org/abs/2103.14438 · HTML: https://mlsanity.com/paper/2103.14438/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 20 references that are in the map.
- First posted on arXiv 26 Mar 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [LSTM Fully Convolutional Networks for Time Series Classification](https://mlsanity.com/paper/1709.05206/) (2017)
- [Multi-Scale Convolutional Neural Networks for Time Series Classification](https://mlsanity.com/paper/1603.06995/) (2016)
- [Self-attention for raw optical Satellite Time Series Classification](https://mlsanity.com/paper/1910.10536/) (2019)
- [Towards a universal neural network encoder for time series](https://mlsanity.com/paper/1805.03908/) (2018)
- [Learning to Exploit Invariances in Clinical Time-Series Data using Sequence Transformer Networks](https://mlsanity.com/paper/1808.06725/) (2018)
- [Encoding Temporal Markov Dynamics in Graph for Visualizing and Mining Time Series](https://mlsanity.com/paper/1610.07273/) (2016)

## Built on by (most cited first, 30 of 58)

- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters](https://mlsanity.com/paper/2308.08469/) (2023)
- [OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensembling](https://mlsanity.com/paper/2309.12659/) (2023)
- [Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey](https://mlsanity.com/paper/2302.02515/) (2023)
- [Medformer: A Multi-Granularity Patching Transformer for Medical Time-Series Classification](https://mlsanity.com/paper/2405.19363/) (2024)
- [Transformers in Time-series Analysis: A Tutorial](https://mlsanity.com/paper/2205.01138/) (2022)
- [Improving Position Encoding of Transformers for Multivariate Time Series Classification](https://mlsanity.com/paper/2305.16642/) (2023)
- [TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders](https://mlsanity.com/paper/2303.00320/) (2023)
- [ConvTimeNet: A Deep Hierarchical Fully Convolutional Model for Multivariate Time Series Analysis](https://mlsanity.com/paper/2403.01493/) (2024)
- [FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification](https://mlsanity.com/paper/2302.09818/) (2023)
- [Mesogeos: A multi-purpose dataset for data-driven wildfire modeling in the Mediterranean](https://mlsanity.com/paper/2306.05144/) (2023)
- [BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics](https://mlsanity.com/paper/2501.18972/) (2025)
- [ShapeFormer: Shapelet Transformer for Multivariate Time Series Classification](https://mlsanity.com/paper/2405.14608/) (2024)
- [Positional Encoding in Transformer-Based Time Series Models: A Survey](https://mlsanity.com/paper/2502.12370/) (2025)
- [Cross-Domain Pre-training with Language Models for Transferable Time Series Representations](https://mlsanity.com/paper/2403.12372/) (2024)
- [A Systematic Review for Transformer-based Long-term Series Forecasting](https://mlsanity.com/paper/2310.20218/) (2023)
- [TimeDRL: Disentangled Representation Learning for Multivariate Time-Series](https://mlsanity.com/paper/2312.04142/) (2023)
- [Advancing Time Series Classification with Multimodal Language Modeling](https://mlsanity.com/paper/2403.12371/) (2024)
- [Sparse Binary Transformers for Multivariate Time Series Modeling](https://mlsanity.com/paper/2308.04637/) (2023)
- [SeqNAS: Neural Architecture Search for Event Sequence Classification](https://mlsanity.com/paper/2401.03246/) (2024)
- [MTS2Graph: Interpretable Multivariate Time Series Classification with Temporal Evolving Graphs](https://mlsanity.com/paper/2306.03834/) (2023)
- [Novel Physics-Based Machine-Learning Models for Indoor Air Quality Approximations](https://mlsanity.com/paper/2308.01438/) (2023)
- [Quantformer: from attention to profit with a quantitative transformer trading strategy](https://mlsanity.com/paper/2404.00424/) (2024)
- [Efficient High-Resolution Time Series Classification via Attention Kronecker Decomposition](https://mlsanity.com/paper/2403.04882/) (2024)
- [Enhancing Transformer Efficiency for Multivariate Time Series Classification](https://mlsanity.com/paper/2203.14472/) (2022)
- [MSW-Transformer: Multi-Scale Shifted Windows Transformer Networks for 12-Lead ECG Classification](https://mlsanity.com/paper/2306.12098/) (2023)
- [Training-Free Time Series Classification via In-Context Reasoning with LLM Agents](https://mlsanity.com/paper/2510.05950/) (2025)
- [Diagnostic Spatio-temporal Transformer with Faithful Encoding](https://mlsanity.com/paper/2305.17149/) (2023)
- [A Reliable Framework for Human-in-the-Loop Anomaly Detection in Time Series](https://mlsanity.com/paper/2405.03234/) (2024)
- [A Survey of Spatio-Temporal EEG data Analysis: from Models to Applications](https://mlsanity.com/paper/2410.08224/) (2024)

## Abstract (arXiv)

Deep learning model (primarily convolutional networks and LSTM) for time series classification has been studied broadly by the community with the wide applications in different domains like healthcare, finance, industrial engineering and IoT. Meanwhile, Transformer Networks recently achieved frontier performance on various natural language processing and computer vision tasks. In this work, we explored a simple extension of the current Transformer Networks with gating, named Gated Transformer Networks (GTN) for the multivariate time series classification problem. With the gating that merges two towers of Transformer which model the channel-wise and step-wise correlations respectively, we show how GTN is naturally and effectively suitable for the multivariate time series classification task. We conduct comprehensive experiments on thirteen dataset with full ablation study. Our results show that GTN is able to achieve competing results with current state-of-the-art deep learning models. We also explored the attention map for the natural interpretability of GTN on time series modeling. Our preliminary results provide a strong baseline for the Transformer Networks on multivariate time series classification task and grounds the foundation for future research.