# Improving Position Encoding of Transformers for Multivariate Time Series Classification

arXiv: https://arxiv.org/abs/2305.16642 · HTML: https://mlsanity.com/paper/2305.16642/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [CoAtNet: Marrying Convolution and Attention for All Data Sizes](https://mlsanity.com/paper/2106.04803/) (2021)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [InceptionTime: Finding AlexNet for Time Series Classification](https://mlsanity.com/paper/1909.04939/) (2019)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels](https://mlsanity.com/paper/1910.13051/) (2019)
- [The UEA multivariate time series classification archive, 2018](https://mlsanity.com/paper/1811.00075/) (2018)
- [Rethinking and Improving Relative Position Encoding for Vision Transformer](https://mlsanity.com/paper/2107.14222/) (2021)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classification](https://mlsanity.com/paper/2012.08791/) (2020)
- [Benchmarking Multivariate Time Series Classification Algorithms](https://mlsanity.com/paper/2007.13156/) (2020)
- [Multivariate LSTM-FCNs for Time Series Classification](https://mlsanity.com/paper/1801.04503/) (2018)
- [HIVE-COTE 2.0: a new meta ensemble for time series classification](https://mlsanity.com/paper/2104.07551/) (2021)
- [Position Information in Transformers: An Overview](https://mlsanity.com/paper/2102.11090/) (2021)
- [Voice2Series: Reprogramming Acoustic Models for Time Series Classification](https://mlsanity.com/paper/2106.09296/) (2021)
- [Improve Transformer Models with Better Relative Position Embeddings](https://mlsanity.com/paper/2009.13658/) (2020)
- [Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey](https://mlsanity.com/paper/2302.02515/) (2023)
- [Gated Transformer Networks for Multivariate Time Series Classification](https://mlsanity.com/paper/2103.14438/) (2021)
- [The Canonical Interval Forest (CIF) Classifier for Time Series Classification](https://mlsanity.com/paper/2008.09172/) (2020)
- [Learning to Remove: Towards Isotropic Pre-trained BERT Embedding](https://mlsanity.com/paper/2104.05274/) (2021)

## Built on by (most cited first, 30 of 45)

- [Is Mamba Effective for Time Series Forecasting?](https://mlsanity.com/paper/2403.11144/) (2024)
- [EEG2Rep: Enhancing Self-supervised EEG Representation Through Informative Masked Inputs](https://mlsanity.com/paper/2402.17772/) (2024)
- [Toward Foundation Model for Multivariate Wearable Sensing of Physiological Signals](https://mlsanity.com/paper/2412.09758/) (2024)
- [Positional Encoding in Transformer-Based Time Series Models: A Survey](https://mlsanity.com/paper/2502.12370/) (2025)
- [DARNet: Dual Attention Refinement Network with Spatiotemporal Construction for Auditory Attention Detection](https://mlsanity.com/paper/2410.11181/) (2024)
- [Look Into the LITE in Deep Learning for Time Series Classification](https://mlsanity.com/paper/2409.02869/) (2024)
- [Series2Vec: Similarity-based Self-supervised Representation Learning for Time Series Classification](https://mlsanity.com/paper/2312.03998/) (2023)
- [ms-Mamba: Multi-scale Mamba for Time-Series Forecasting](https://mlsanity.com/paper/2504.07654/) (2025)
- [Transformer with Leveraged Masked Autoencoder for video-based Pain Assessment](https://mlsanity.com/paper/2409.05088/) (2024)
- [MONSTER: Monash Scalable Time Series Evaluation Repository](https://mlsanity.com/paper/2502.15122/) (2025)
- [POCKET: Pruning Random Convolution Kernels for Time Series Classification from a Feature Selection Perspective](https://mlsanity.com/paper/2309.08499/) (2023)
- [Improving the Evaluation and Actionability of Explanation Methods for Multivariate Time Series Classification](https://mlsanity.com/paper/2406.12507/) (2024)
- [The Multiverse of Time Series Machine Learning: an Archive for Multivariate Time Series Classification](https://mlsanity.com/paper/2603.20352/) (2026)
- [Survey and Taxonomy: The Role of Data-Centric AI in Transformer-Based Time Series Forecasting](https://mlsanity.com/paper/2407.19784/) (2024)
- [NILMFormer: Non-Intrusive Load Monitoring that Accounts for Non-Stationarity](https://mlsanity.com/paper/2506.05880/) (2025)
- [VSFormer: Value and Shape-Aware Transformer with Prior-Enhanced Self-Attention for Multivariate Time Series Classification](https://mlsanity.com/paper/2412.16515/) (2024)
- [FIC-TSC: Learning Time Series Classification with Fisher Information Constraint](https://mlsanity.com/paper/2505.06114/) (2025)
- [A Standardized Benchmark for Skeleton-Based Rehabilitation Assessment Using Deep Learning](https://mlsanity.com/paper/2507.21018/) (2025)
- [Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration](https://mlsanity.com/paper/2510.19579/) (2025)
- [Stochastic Parroting in Temporal Attention -- Regulating the Diagonal Sink](https://mlsanity.com/paper/2602.10956/) (2026)
- [Convolutional and Deep Learning based techniques for Time Series Ordinal Classification](https://mlsanity.com/paper/2306.10084/) (2023)
- [TFBEST: Dual-Aspect Transformer with Learnable Positional Encoding for Failure Prediction](https://mlsanity.com/paper/2309.02641/) (2023)
- [ExChanGeAI: An End-to-End Platform and Efficient Foundation Model for Electrocardiogram Analysis and Fine-tuning](https://mlsanity.com/paper/2503.13570/) (2025)
- [Scaling Transformers for Time Series Forecasting: Do Pretrained Large Models Outperform Small-Scale Alternatives?](https://mlsanity.com/paper/2507.02907/) (2025)
- [TimeSenCLIP: A Time Series Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2508.11919/) (2025)
- [DyWPE: Signal-Aware Dynamic Wavelet Positional Encoding for Time Series Transformers](https://mlsanity.com/paper/2509.14640/) (2025)
- [Data-Driven Dynamic Parameter Learning of manipulator robots](https://mlsanity.com/paper/2512.08767/) (2025)
- [Data-Local Autonomous LLM-Guided Neural Architecture Search for Multiclass Multimodal Time-Series Classification](https://mlsanity.com/paper/2603.15939/) (2026)
- [Transformer-based Named Entity Recognition in Construction Supply Chain Risk Management in Australia](https://mlsanity.com/paper/2311.13755/) (2023)
- [Human Brain Exhibits Distinct Patterns When Listening to Fake Versus Real Audio: Preliminary Evidence](https://mlsanity.com/paper/2402.14982/) (2024)

## Abstract (arXiv)

Transformers have demonstrated outstanding performance in many applications of deep learning. When applied to time series data, transformers require effective position encoding to capture the ordering of the time series data. The efficacy of position encoding in time series analysis is not well-studied and remains controversial, e.g., whether it is better to inject absolute position encoding or relative position encoding, or a combination of them. In order to clarify this, we first review existing absolute and relative position encoding methods when applied in time series classification. We then proposed a new absolute position encoding method dedicated to time series data called time Absolute Position Encoding (tAPE). Our new method incorporates the series length and input embedding dimension in absolute position encoding. Additionally, we propose computationally Efficient implementation of Relative Position Encoding (eRPE) to improve generalisability for time series. We then propose a novel multivariate time series classification (MTSC) model combining tAPE/eRPE and convolution-based input encoding named ConvTran to improve the position and data embedding of time series data. The proposed absolute and relative position encoding methods are simple and efficient. They can be easily integrated into transformer blocks and used for downstream tasks such as forecasting, extrinsic regression, and anomaly detection. Extensive experiments on 32 multivariate time-series datasets show that our model is significantly more accurate than state-of-the-art convolution and transformer-based models. Code and models are open-sourced at urlhttps://github.com/Navidfoumani/ConvTran.