# Enhancing Transformer Efficiency for Multivariate Time Series Classification

arXiv: https://arxiv.org/abs/2203.14472 · HTML: https://mlsanity.com/paper/2203.14472/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 44 references that are in the map.
- First posted on arXiv 28 Mar 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [InceptionTime: Finding AlexNet for Time Series Classification](https://mlsanity.com/paper/1909.04939/) (2019)
- [Bottleneck Transformers for Visual Recognition](https://mlsanity.com/paper/2101.11605/) (2021)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels](https://mlsanity.com/paper/1910.13051/) (2019)
- [Time-Series Anomaly Detection Service at Microsoft](https://mlsanity.com/paper/1906.03821/) (2019)
- [Benchmarking Multivariate Time Series Classification Algorithms](https://mlsanity.com/paper/2007.13156/) (2020)
- [Attend and Diagnose: Clinical Time Series Analysis using Attention Models](https://mlsanity.com/paper/1711.03905/) (2017)
- [Gated Transformer Networks for Multivariate Time Series Classification](https://mlsanity.com/paper/2103.14438/) (2021)
- [Highly comparative feature-based time-series classification](https://mlsanity.com/paper/1401.3531/) (2014)
- [A review on distance based time series classification](https://mlsanity.com/paper/1806.04509/) (2018)
- [BERTSurv: BERT-Based Survival Models for Predicting Outcomes of Trauma Patients](https://mlsanity.com/paper/2103.10928/) (2021)
- [Learning to Exploit Invariances in Clinical Time-Series Data using Sequence Transformer Networks](https://mlsanity.com/paper/1808.06725/) (2018)

## Built on by (most cited first, 4 of 4)

- [TRAM: Benchmarking Temporal Reasoning for Large Language Models](https://mlsanity.com/paper/2310.00835/) (2023)
- [Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Models](https://mlsanity.com/paper/2312.17661/) (2023)
- [RUPBench: Benchmarking Reasoning Under Perturbations for Robustness Evaluation in Large Language Models](https://mlsanity.com/paper/2406.11020/) (2024)
- [Causal and Local Correlations Based Network for Multivariate Time Series Classification](https://mlsanity.com/paper/2411.18008/) (2024)

## Abstract (arXiv)

Most current multivariate time series (MTS) classification algorithms focus on improving the predictive accuracy. However, for large-scale (either high-dimensional or long-sequential) time series (TS) datasets, there is an additional consideration: to design an efficient network architecture to reduce computational costs such as training time and memory footprint. In this work we propose a methodology based on module-wise pruning and Pareto analysis to investigate the relationship between model efficiency and accuracy, as well as its complexity. Comprehensive experiments on benchmark MTS datasets illustrate the effectiveness of our method.