# Positional Encoding in Transformer-Based Time Series Models: A Survey

arXiv: https://arxiv.org/abs/2502.12370 · HTML: https://mlsanity.com/paper/2502.12370/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 84 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [Time Series Forecasting With Deep Learning: A Survey](https://mlsanity.com/paper/2004.13408/) (2020)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [Are Transformers universal approximators of sequence-to-sequence functions?](https://mlsanity.com/paper/1912.10077/) (2019)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy](https://mlsanity.com/paper/2110.02642/) (2021)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [Monash Time Series Forecasting Archive](https://mlsanity.com/paper/2105.06643/) (2021)

## Built on by (most cited first, 14 of 14)

- [Diffusion Models for Time Series Forecasting: A Survey](https://mlsanity.com/paper/2507.14507/) (2025)
- [Network Anomaly Detection for IoT Using Hyperdimensional Computing on NSL-KDD](https://mlsanity.com/paper/2503.03031/) (2025)
- [Unsupervised Time-Series Signal Analysis with Autoencoders and Vision Transformers: A Review of Architectures and Applications](https://mlsanity.com/paper/2504.16972/) (2025)
- [DyWPE: Signal-Aware Dynamic Wavelet Positional Encoding for Time Series Transformers](https://mlsanity.com/paper/2509.14640/) (2025)
- [Securing Time Integrity in Energy IoT Against Clock Drift and Y2K38 Failures](https://mlsanity.com/paper/2601.23147/) (2026)
- [BiTimeCrossNet: Time-Aware Self-Supervised Learning for Pediatric Sleep](https://mlsanity.com/paper/2602.02769/) (2026)
- [T2SGrid: Temporal-to-Spatial Gridification for Video Temporal Grounding](https://mlsanity.com/paper/2603.06973/) (2026)
- [Multi-vessel Interaction-Aware Trajectory Prediction and Collision Risk Assessment](https://mlsanity.com/paper/2509.01836/) (2025)
- [BSAT: B-Spline Adaptive Tokenizer for Long-Term Time Series Forecasting](https://mlsanity.com/paper/2601.00698/) (2026)
- [From Hawkes Processes to Attention: Time-Modulated Mechanisms for Event Sequences](https://mlsanity.com/paper/2601.09220/) (2026)
- [Attention-Enhanced Graph Filtering for False Data Injection Attack Detection and Localization](https://mlsanity.com/paper/2601.18981/) (2026)
- [WaveFormer: Wavelet Embedding Transformer for Biomedical Signals](https://mlsanity.com/paper/2602.12189/) (2026)
- [An Empirical Audit of Input Encoders for Multi-Channel Signal Transformers](https://mlsanity.com/paper/2606.04752/) (2026)
- [Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecasting](https://mlsanity.com/paper/2610.19603/) (2026)

## Abstract (arXiv)

Recent advancements in transformer-based models have greatly improved time series analysis, providing robust solutions for tasks such as forecasting, anomaly detection, and classification. A crucial element of these models is positional encoding, which allows transformers to capture the intrinsic sequential nature of time series data. This survey systematically examines existing techniques for positional encoding in transformer-based time series models. We investigate a variety of methods, including fixed, learnable, relative, and hybrid approaches, and evaluate their effectiveness in different time series classification tasks. Our findings indicate that data characteristics like sequence length, signal complexity, and dimensionality significantly influence method effectiveness. Advanced positional encoding methods exhibit performance gains in terms of prediction accuracy, however, they come at the cost of increased computational complexity. Furthermore, we outline key challenges and suggest potential research directions to enhance positional encoding strategies. By delivering a comprehensive overview and quantitative benchmarking, this survey intends to assist researchers and practitioners in selecting and designing effective positional encoding methods for transformer-based time series models.