# Temporal Convolutional Attention-based Network For Sequence Modeling

arXiv: https://arxiv.org/abs/2002.12530 · HTML: https://mlsanity.com/paper/2002.12530/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2020.

## Built on by (most cited first, 8 of 8)

- [Parallel Spatio-Temporal Attention-Based TCN for Multivariate Time Series Prediction](https://mlsanity.com/paper/2203.00971/) (2022)
- [MixSeq: Connecting Macroscopic Time Series Forecasting with Microscopic Time Series Data](https://mlsanity.com/paper/2110.14354/) (2021)
- [Towards better traffic volume estimation: Jointly addressing the underdetermination and nonequilibrium problems with correlation-adaptive GNNs](https://mlsanity.com/paper/2303.05660/) (2023)
- [Learning Differential Operators for Interpretable Time Series Modeling](https://mlsanity.com/paper/2209.01491/) (2022)
- [Step Counting with Attention-based LSTM](https://mlsanity.com/paper/2211.13114/) (2022)
- [WiFi-TCN: Temporal Convolution for Human Interaction Recognition based on WiFi signal](https://mlsanity.com/paper/2305.18211/) (2023)
- [TCNCA: Temporal Convolution Network with Chunked Attention for Scalable Sequence Processing](https://mlsanity.com/paper/2312.05605/) (2023)
- [NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding](https://mlsanity.com/paper/2312.07507/) (2023)

## Abstract (arXiv)

With the development of feed-forward models, the default model for sequence modeling has gradually evolved to replace recurrent networks. Many powerful feed-forward models based on convolutional networks and attention mechanism were proposed and show more potential to handle sequence modeling tasks. We wonder that is there an architecture that can not only achieve an approximate substitution of recurrent network, but also absorb the advantages of feed-forward models. So we propose an exploratory architecture referred to Temporal Convolutional Attention-based Network (TCAN) which combines temporal convolutional network and attention mechanism. TCAN includes two parts, one is Temporal Attention (TA) which captures relevant features inside the sequence, the other is Enhanced Residual (ER) which extracts shallow layer's important information and transfers to deep layers. We improve the state-of-the-art results of bpc/perplexity to 30.28 on word-level PTB, 1.092 on character-level PTB, and 9.20 on WikiText-2.