mlsanityData to 7 Oct 2026

Transformers in Time-series Analysis: A Tutorial

arXiv 2205.01138 ↗First posted 28 Apr 2022cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

Transformer architecture has widespread applications, particularly in Natural Language Processing and computer vision. Recently Transformers have been employed in various aspects of time-series analysis. This tutorial provides an overview of the Transformer architecture, its applications, and a collection of examples from recent research papers in time-series analysis. We delve into an explanation of the core components of the Transformer, including the self-attention mechanism, positional encoding, multi-head, and encoder/decoder. Several enhancements to the initial, Transformer architecture are highlighted to tackle time-series tasks. The tutorial also provides best practices and techniques to overcome the challenge of effectively training Transformers for time-series analysis.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 71 of its 123 references that are in the map. First posted on arXiv 28 Apr 2022.

Builds on

71 of 123 references.

References · 71

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016Layer Normalizationcited by 5,579
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2014Going Deeper with Convolutionscited by 4,231
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  9. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  10. 2018Deep contextualized word representationscited by 2,285
  11. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  12. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
38 more
  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  2. 2020Reformer: The Efficient Transformercited by 1,658
  3. 2020Rethinking Attention with Performerscited by 1,584
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  6. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  7. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,004
  8. 2020On Layer Normalization in the Transformer Architecturecited by 906
  9. 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 869
  10. 2020Efficient Transformers: A Surveycited by 784
  11. 2019Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecastingcited by 748
  12. 2016Language Modeling with Gated Convolutional Networkscited by 688
  13. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  14. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 606
  15. 2017Convolutional Sequence to Sequence Learningcited by 550
  16. 2016Recurrent Neural Networks for Multivariate Time Series with Missing Valuescited by 548
  17. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  18. 2018Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Networkcited by 415
  19. 2021A Survey of Transformerscited by 399
  20. 2021Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancycited by 387
  21. 2019CSPNet: A New Backbone that can Enhance Learning Capability of CNNcited by 381
  22. 2016Time Series Classification from Scratch with Deep Neural Networks: A Strong Baselinecited by 358
  23. 2019Axial Attention in Multidimensional Transformerscited by 350
  24. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  25. 2017Dilated Residual Networkscited by 293
  26. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  27. 2015A Critical Review of Recurrent Neural Networks for Sequence Learningcited by 223
  28. 2018A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decaycited by 223
  29. 2020Understanding the Difficulty of Training Transformerscited by 210
  30. 2019AliGraph: A Comprehensive Graph Neural Network Platformcited by 152
  31. 2018Recent Advances in Autoencoder-Based Representation Learningcited by 130
  32. 2020Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Casecited by 129
  33. 2017Attend and Diagnose: Clinical Time Series Analysis using Attention Modelscited by 105
  34. 2018Training Tips for the Transformer Modelcited by 85
  35. 2020Tabular Transformers for Modeling Multivariate Time Seriescited by 77
  36. 2018Temporal Convolutional Neural Network for the Classification of Satellite Image Time Seriescited by 76
  37. 2019Self-attention for raw optical Satellite Time Series Classificationcited by 75
  38. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 20; 2025: 14; 2026: 7.

45 papers.

Sort:

By area:

  1. 2023LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasterscited by 127
  2. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  3. 2024Is Mamba Effective for Time Series Forecasting?cited by 110
  4. 2024Transforming Multidimensional Time Series into Interpretable Event Sequences for Advanced Data Miningcited by 47
  5. 2025Primus: Enforcing Attention Usage for 3D Medical Image Segmentationcited by 22
  6. 2023Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Reviewcited by 20
  7. 2024Mamba-in-Mamba: Centralized Mamba-Cross-Scan in Tokenized Mamba Model for Hyperspectral Image Classificationcited by 18
  8. 2024HARMamba: Efficient and Lightweight Wearable Sensor Human Activity Recognition Based on Bidirectional Mambacited by 11
  9. 2025ms-Mamba: Multi-scale Mamba for Time-Series Forecastingcited by 7
  10. 2023Building Flexible, Scalable, and Machine Learning-ready Multimodal Oncology Datasetscited by 6
  11. 2024A Survey of Transformer Enabled Time Series Synthesiscited by 5
  12. 2024Domain Adaptation for Time series Transformers using One-step fine-tuningcited by 3
33 more
  1. 2024Survey and Taxonomy: The Role of Data-Centric AI in Transformer-Based Time Series Forecastingcited by 3
  2. 2024Early Prediction of Causes (not Effects) in Healthcare by Long-Term Clinical Time Series Forecastingcited by 3
  3. 2025time2time: Causal Intervention in Hidden States to Simulate Rare Events in Time Series Foundation Modelscited by 3
  4. 2023Efficient Scopeformer: Towards Scalable and Rich Feature Extraction for Intracranial Hemorrhage Detectioncited by 2
  5. 2024TransfoRhythm: A Transformer Architecture Conductive to Blood Pressure Estimation via Solo PPG Signal Capturingcited by 2
  6. 2024Embedding-based Multimodal Learning on Pan-Squamous Cell Carcinomas for Improved Survival Outcomescited by 2
  7. 2024Transfer Learning on Transformers for Building Energy Consumption Forecasting -- A Comparative Studycited by 2
  8. 2024Text Classification: Neural Networks VS Machine Learning Models VS Pre-trained Modelscited by 2
  9. 2024Self-Normalizing Foundation Model for Enhanced Multi-Omics Data Analysis in Oncologycited by 1
  10. 2024A Systematic Evaluation of Generated Time Series and Their Effects in Self-Supervised Pretrainingcited by 1
  11. 2024Integrating the Expected Future in Load Forecasts with Contextually Enhanced Transformer Modelscited by 1
  12. 2025A Unified Hyperparameter Optimization Pipeline for Transformer-Based Time Series Forecasting Modelscited by 1
  13. 2025Mitigating Data Scarcity in Time Series Analysis: A Foundation Model with Series-Symbol Data Generationcited by 1
  14. 2025Enhancing Time Series Forecasting with Fuzzy Attention-Integrated Transformerscited by 1
  15. 2025Benchmarking Unsupervised Strategies for Anomaly Detection in Multivariate Time Seriescited by 1
  16. 2026Regime-aware financial volatility forecasting via in-context learningcited by 1
  17. 2026TRACE: A Temporal Conditional Estimation for Multimodal Time Series Foundation Modelscited by 1
  18. 2024Robustness Evaluation of Machine Learning Models for Robot Arm Action Recognition in Noisy Environmentscited by 0
  19. 2024An end-to-end deep learning pipeline to derive blood input with partial volume corrections for automated parametric brain PET mappingcited by 0
  20. 2024Generating Synthetic Time Series Data for Cyber-Physical Systemscited by 0
  21. 2024TEL'M: Test and Evaluation of Language Modelscited by 0
  22. 2025PA-CFL: Privacy-Adaptive Clustered Federated Learning for Transformer-Based Sales Forecasting on Heterogeneous Retail Datacited by 0
  23. 2025Transformer-Based Inverse Microrheology for Experimental Mechanics at Ultra-High Strain Ratescited by 0
  24. 2025Scaling Transformers for Time Series Forecasting: Do Pretrained Large Models Outperform Small-Scale Alternatives?cited by 0
  25. 2025Explainable AI in Genomics: Transcription Factor Binding Site Prediction with Mixture of Expertscited by 0
  26. 2025Wavelet-SARIMA-Transformer: A Hybrid Model for Rainfall Forecastingcited by 0
  27. 2025Enhancing Nuclear Reactor Core Simulation through Data-Based Surrogate Modelscited by 0
  28. 2025Prediction of Herd Life in Dairy Cows Using Multi-Head Attention Transformerscited by 0
  29. 2026Scaled Dot-Product Attention implements projection of inputs onto a common surfacecited by 0
  30. 2026Central limit theorems for the outputs of fully convolutional neural networks with time series inputcited by 0
  31. 2026Transformer autoencoder with local attention for sparse and irregular time series with application on risk estimationcited by 0
  32. 2026Leveraging Physiological Signals to Predict Exam Outcomes with Machine Learningcited by 0
  33. 2026Characterizing and Mitigating the Effects of Device Temperature on RF Fingerprinting Accuracycited by 0