# Transformers in Time-series Analysis: A Tutorial

arXiv: https://arxiv.org/abs/2205.01138 · HTML: https://mlsanity.com/paper/2205.01138/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 71 of its 123 references that are in the map.
- First posted on arXiv 28 Apr 2022.

## Builds on (most cited first, 30 of 71)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Efficient Content-Based Sparse Attention with Routing Transformers](https://mlsanity.com/paper/2003.05997/) (2020)
- [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018)

## Built on by (most cited first, 30 of 46)

- [LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters](https://mlsanity.com/paper/2308.08469/) (2023)
- [Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review](https://mlsanity.com/paper/2403.02469/) (2024)
- [Is Mamba Effective for Time Series Forecasting?](https://mlsanity.com/paper/2403.11144/) (2024)
- [Transforming Multidimensional Time Series into Interpretable Event Sequences for Advanced Data Mining](https://mlsanity.com/paper/2409.14327/) (2024)
- [Primus: Enforcing Attention Usage for 3D Medical Image Segmentation](https://mlsanity.com/paper/2503.01835/) (2025)
- [Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review](https://mlsanity.com/paper/2303.06471/) (2023)
- [Mamba-in-Mamba: Centralized Mamba-Cross-Scan in Tokenized Mamba Model for Hyperspectral Image Classification](https://mlsanity.com/paper/2405.12003/) (2024)
- [HARMamba: Efficient and Lightweight Wearable Sensor Human Activity Recognition Based on Bidirectional Mamba](https://mlsanity.com/paper/2403.20183/) (2024)
- [ms-Mamba: Multi-scale Mamba for Time-Series Forecasting](https://mlsanity.com/paper/2504.07654/) (2025)
- [Building Flexible, Scalable, and Machine Learning-ready Multimodal Oncology Datasets](https://mlsanity.com/paper/2310.01438/) (2023)
- [A Survey of Transformer Enabled Time Series Synthesis](https://mlsanity.com/paper/2406.02322/) (2024)
- [time2time: Causal Intervention in Hidden States to Simulate Rare Events in Time Series Foundation Models](https://mlsanity.com/paper/2509.05801/) (2025)
- [Domain Adaptation for Time series Transformers using One-step fine-tuning](https://mlsanity.com/paper/2401.06524/) (2024)
- [Survey and Taxonomy: The Role of Data-Centric AI in Transformer-Based Time Series Forecasting](https://mlsanity.com/paper/2407.19784/) (2024)
- [Early Prediction of Causes (not Effects) in Healthcare by Long-Term Clinical Time Series Forecasting](https://mlsanity.com/paper/2408.03816/) (2024)
- [Efficient Scopeformer: Towards Scalable and Rich Feature Extraction for Intracranial Hemorrhage Detection](https://mlsanity.com/paper/2302.00220/) (2023)
- [TransfoRhythm: A Transformer Architecture Conductive to Blood Pressure Estimation via Solo PPG Signal Capturing](https://mlsanity.com/paper/2404.15352/) (2024)
- [Embedding-based Multimodal Learning on Pan-Squamous Cell Carcinomas for Improved Survival Outcomes](https://mlsanity.com/paper/2406.08521/) (2024)
- [Transfer Learning on Transformers for Building Energy Consumption Forecasting -- A Comparative Study](https://mlsanity.com/paper/2410.14107/) (2024)
- [Text Classification: Neural Networks VS Machine Learning Models VS Pre-trained Models](https://mlsanity.com/paper/2412.21022/) (2024)
- [TRACE: A Temporal Conditional Estimation for Multimodal Time Series Foundation Models](https://mlsanity.com/paper/2606.06285/) (2026)
- [Self-Normalizing Foundation Model for Enhanced Multi-Omics Data Analysis in Oncology](https://mlsanity.com/paper/2405.08226/) (2024)
- [A Systematic Evaluation of Generated Time Series and Their Effects in Self-Supervised Pretraining](https://mlsanity.com/paper/2408.07869/) (2024)
- [Integrating the Expected Future in Load Forecasts with Contextually Enhanced Transformer Models](https://mlsanity.com/paper/2409.05884/) (2024)
- [A Unified Hyperparameter Optimization Pipeline for Transformer-Based Time Series Forecasting Models](https://mlsanity.com/paper/2501.01394/) (2025)
- [Mitigating Data Scarcity in Time Series Analysis: A Foundation Model with Series-Symbol Data Generation](https://mlsanity.com/paper/2502.15466/) (2025)
- [FANTF: Fuzzy attention network based transformers for time series analysis](https://mlsanity.com/paper/2504.00070/) (2025)
- [Benchmarking Unsupervised Strategies for Anomaly Detection in Multivariate Time Series](https://mlsanity.com/paper/2506.20574/) (2025)
- [Scaling Transformers for Time Series Forecasting: Do Pretrained Large Models Outperform Small-Scale Alternatives?](https://mlsanity.com/paper/2507.02907/) (2025)
- [Regime-aware financial volatility forecasting via in-context learning](https://mlsanity.com/paper/2603.10299/) (2026)

## Abstract (arXiv)

Transformer architecture has widespread applications, particularly in Natural Language Processing and computer vision. Recently Transformers have been employed in various aspects of time-series analysis. This tutorial provides an overview of the Transformer architecture, its applications, and a collection of examples from recent research papers in time-series analysis. We delve into an explanation of the core components of the Transformer, including the self-attention mechanism, positional encoding, multi-head, and encoder/decoder. Several enhancements to the initial, Transformer architecture are highlighted to tackle time-series tasks. The tutorial also provides best practices and techniques to overcome the challenge of effectively training Transformers for time-series analysis.