mlsanityData to 7 Oct 2026

Rethinking Attention Mechanism in Time Series Classification

arXiv 2207.07564 ↗First posted 14 Jul 2022cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Attention-based models have been widely used in many areas, such as computer vision and natural language processing. However, relevant applications in time series classification (TSC) have not been explored deeply yet, causing a significant number of TSC algorithms still suffer from general problems of attention mechanism, like quadratic complexity. In this paper, we promote the efficiency and performance of the attention mechanism by proposing our flexible multi-head linear attention (FMLA), which enhances locality awareness by layer-wise interactions with deformable convolutional blocks and online knowledge distillation. What's more, we propose a simple but effective mask mechanism that helps reduce the noise influence in time series and decrease the redundancy of the proposed FMLA by masking some positions of each given series proportionally. To stabilize this mechanism, samples are forwarded through the model with random mask layers several times and their outputs are aggregated to teach the same model with regular mask layers. We conduct extensive experiments on 85 UCR2018 datasets to compare our algorithm with 11 well-known ones and the results show that our algorithm has comparable performance in terms of top-1 accuracy. We also compare our model with three Transformer-based models with respect to the floating-point operations per second and number of parameters and find that our algorithm achieves significantly better efficiency with lower complexity.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 50 of its 72 references that are in the map. First posted on arXiv 14 Jul 2022.

Builds on

50 of 72 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  12. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
38 more
  1. 2017Deformable Convolutional Networkscited by 1,481
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  3. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  5. 2021Multiscale Vision Transformerscited by 896
  6. 2020Efficient Transformers: A Surveycited by 784
  7. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  8. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 606
  9. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  10. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  11. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  12. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  13. 2020A Transformer-based Framework for Multivariate Time Series Representation Learningcited by 449
  14. 2018The UCR Time Series Archivecited by 421
  15. 2019InceptionTime: Finding AlexNet for Time Series Classificationcited by 386
  16. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  17. 2019ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernelscited by 336
  18. 2022Vision Transformer with Deformable Attentioncited by 293
  19. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  20. 2020MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classificationcited by 186
  21. 2021Conformer: Local Features Coupling Global Representations for Visual Recognitioncited by 164
  22. 2020Benchmarking Multivariate Time Series Classification Algorithmscited by 145
  23. 2017LSTM Fully Convolutional Networks for Time Series Classificationcited by 135
  24. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  25. 2021HIVE-COTE 2.0: a new meta ensemble for time series classificationcited by 118
  26. 2021MultiRocket: Multiple pooling operators and transformations for fast and effective time series classificationcited by 89
  27. 2021On the Integration of Self-Attention and Convolutioncited by 74
  28. 2019TS-CHIEF: A Scalable and Accurate Forest Algorithm for Time Series Classificationcited by 67
  29. 2021On the Connection between Local Attention and Dynamic Depth-wise Convolutioncited by 61
  30. 2021Poolingformer: Long Document Modeling with Pooling Attentioncited by 60
  31. 2020Multi-Head Attention: Collaborate Instead of Concatenatecited by 56
  32. 2018Proximity Forest: An effective and scalable distance-based classifier for time seriescited by 46
  33. 2021Mask Attention Networks: Rethinking and Strengthen Transformercited by 28
  34. 2020RTFN: A Robust Temporal Feature Network for Time Series Classificationcited by 11
  35. 2019Financial series prediction using Attention LSTMcited by 7
  36. 2020Gated Recurrent Fusion with Joint Training Framework for Robust End-to-End Speech Recognitioncited by 7
  37. 2020Residual Attention Net for Superior Cross-Domain Time Sequence Modelingcited by 2
  38. 2019Timage -- A Robust Time Series Classification Pipelinecited by 1

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 1; 2025: 2.

5 papers.

  1. 2023Deep Learning for Time Series Classification and Extrinsic Regression: A Current Surveycited by 68
  2. 2022FECAM: Frequency Enhanced Channel Attention Mechanism for Time Series Forecastingcited by 15
  3. 2024Con4m: Context-aware Consistency Learning Framework for Segmented Time Series Classificationcited by 2
  4. 2025Implet: A Post-hoc Subsequence Explainer for Time Series Modelscited by 0
  5. 2025SAFE-D: A Spatiotemporal Detection Framework for Abnormal Driving Among Parkinson's Disease-like Driverscited by 0