# Enhancing Multivariate Time Series Classifiers through Self-Attention and Relative Positioning Infusion

arXiv: https://arxiv.org/abs/2302.06683 · HTML: https://mlsanity.com/paper/2302.06683/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 80 references that are in the map.
- First posted on arXiv 13 Feb 2023.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [InceptionTime: Finding AlexNet for Time Series Classification](https://mlsanity.com/paper/1909.04939/) (2019)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels](https://mlsanity.com/paper/1910.13051/) (2019)
- [The UEA multivariate time series classification archive, 2018](https://mlsanity.com/paper/1811.00075/) (2018)
- [Rethinking Positional Encoding in Language Pre-training](https://mlsanity.com/paper/2006.15595/) (2020)
- [MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classification](https://mlsanity.com/paper/2012.08791/) (2020)
- [HIVE-COTE 2.0: a new meta ensemble for time series classification](https://mlsanity.com/paper/2104.07551/) (2021)
- [Should we really use post-hoc tests based on mean-ranks?](https://mlsanity.com/paper/1505.02288/) (2015)
- [Position Information in Transformers: An Overview](https://mlsanity.com/paper/2102.11090/) (2021)
- [Learning to Diagnose with LSTM Recurrent Neural Networks](https://mlsanity.com/paper/1511.03677/) (2015)
- [Multi-Scale Convolutional Neural Networks for Time Series Classification](https://mlsanity.com/paper/1603.06995/) (2016)
- [Voice2Series: Reprogramming Acoustic Models for Time Series Classification](https://mlsanity.com/paper/2106.09296/) (2021)
- [MultiRocket: Multiple pooling operators and transformations for fast and effective time series classification](https://mlsanity.com/paper/2102.00457/) (2021)
- [Learning to score the figure skating sports videos](https://mlsanity.com/paper/1802.02774/) (2018)
- [Improve Transformer Models with Better Relative Position Embeddings](https://mlsanity.com/paper/2009.13658/) (2020)
- [The Pros and Cons: Rank-aware Temporal Attention for Skill Determination in Long Videos](https://mlsanity.com/paper/1812.05538/) (2018)
- [TS-CHIEF: A Scalable and Accurate Forest Algorithm for Time Series Classification](https://mlsanity.com/paper/1906.10329/) (2019)
- [Gated Transformer Networks for Multivariate Time Series Classification](https://mlsanity.com/paper/2103.14438/) (2021)
- [Omni-Scale CNNs: a simple and effective kernel size configuration for time series classification](https://mlsanity.com/paper/2002.10061/) (2020)

## Built on by (most cited first, 1 of 1)

- [Advancing multivariate time series similarity assessment: an integrated computational approach](https://mlsanity.com/paper/2403.11044/) (2024)

## Abstract (arXiv)

Time Series Classification (TSC) is an important and challenging task for many visual computing applications. Despite the extensive range of methods developed for TSC, relatively few utilized Deep Neural Networks (DNNs). In this paper, we propose two novel attention blocks (Global Temporal Attention and Temporal Pseudo-Gaussian augmented Self-Attention) that can enhance deep learning-based TSC approaches, even when such approaches are designed and optimized for a specific dataset or task. We validate this claim by evaluating multiple state-of-the-art deep learning-based TSC models on the University of East Anglia (UEA) benchmark, a standardized collection of 30 Multivariate Time Series Classification (MTSC) datasets. We show that adding the proposed attention blocks improves base models' average accuracy by up to 3.6%. Additionally, the proposed TPS block uses a new injection module to include the relative positional information in transformers. As a standalone unit with less computational complexity, it enables TPS to perform better than most of the state-of-the-art DNN-based TSC methods. The source codes for our experimental setups and proposed attention blocks are made publicly available.