# TCCT: Tightly-Coupled Convolutional Transformer on Time Series Forecasting

arXiv: https://arxiv.org/abs/2108.12784 · HTML: https://mlsanity.com/paper/2108.12784/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 29 Aug 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [CSPNet: A New Backbone that can Enhance Learning Capability of CNN](https://mlsanity.com/paper/1911.11929/) (2019)
- [A Multi-Horizon Quantile Recurrent Forecaster](https://mlsanity.com/paper/1711.11053/) (2017)
- [You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection](https://mlsanity.com/paper/2106.00666/) (2021)
- [Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case](https://mlsanity.com/paper/2001.08317/) (2020)
- [Attend and Diagnose: Clinical Time Series Analysis using Attention Models](https://mlsanity.com/paper/1711.03905/) (2017)
- [Lightweight Temporal Self-Attention for Classifying Satellite Image Time Series](https://mlsanity.com/paper/2007.00586/) (2020)
- [Long-term Forecasting using Higher Order Tensor RNNs](https://mlsanity.com/paper/1711.00073/) (2017)
- [Dual-stream Network for Visual Recognition](https://mlsanity.com/paper/2105.14734/) (2021)
- [PP-YOLOv2: A Practical Object Detector](https://mlsanity.com/paper/2104.10419/) (2021)
- [Fully Learnable Deep Wavelet Transform for Unsupervised Monitoring of High-Frequency Time Series](https://mlsanity.com/paper/2105.00899/) (2021)
- [Deep Factors with Gaussian Processes for Forecasting](https://mlsanity.com/paper/1812.00098/) (2018)
- [Seq-U-Net: A One-Dimensional Causal U-Net for Efficient Sequence Modelling](https://mlsanity.com/paper/1911.06393/) (2019)
- [Effective Bayesian Modeling of Groups of Related Count Time Series](https://mlsanity.com/paper/1405.3738/) (2014)

## Built on by (most cited first, 8 of 8)

- [Transformers in Time-series Analysis: A Tutorial](https://mlsanity.com/paper/2205.01138/) (2022)
- [A Systematic Review for Transformer-based Long-term Series Forecasting](https://mlsanity.com/paper/2310.20218/) (2023)
- [Geometry Attention Transformer with Position-aware LSTMs for Image Captioning](https://mlsanity.com/paper/2110.00335/) (2021)
- [Survey and Taxonomy: The Role of Data-Centric AI in Transformer-Based Time Series Forecasting](https://mlsanity.com/paper/2407.19784/) (2024)
- [TLNets: Transformation Learning Networks for long-range time-series prediction](https://mlsanity.com/paper/2305.15770/) (2023)
- [Towards smaller, faster decoder-only transformers: Architectural variants and their implications](https://mlsanity.com/paper/2404.14462/) (2024)
- [ADAT: Time-Series-Aware Adaptive Transformer Architecture for Sign Language Translation](https://mlsanity.com/paper/2504.11942/) (2025)
- [Spatially-Enhanced Temporal Fusion Transformer: Interpretable Multi-Output Prediction for Parametric Dynamical Systems with Time-Varying Inputs](https://mlsanity.com/paper/2505.00473/) (2025)

## Abstract (arXiv)

Time series forecasting is essential for a wide range of real-world applications. Recent studies have shown the superiority of Transformer in dealing with such problems, especially long sequence time series input(LSTI) and long sequence time series forecasting(LSTF) problems. To improve the efficiency and enhance the locality of Transformer, these studies combine Transformer with CNN in varying degrees. However, their combinations are loosely-coupled and do not make full use of CNN. To address this issue, we propose the concept of tightly-coupled convolutional Transformer(TCCT) and three TCCT architectures which apply transformed CNN architectures into Transformer: (1) CSPAttention: through fusing CSPNet with self-attention mechanism, the computation cost of self-attention mechanism is reduced by 30% and the memory usage is reduced by 50% while achieving equivalent or beyond prediction accuracy. (2) Dilated causal convolution: this method is to modify the distilling operation proposed by Informer through replacing canonical convolutional layers with dilated causal convolutional layers to gain exponentially receptive field growth. (3) Passthrough mechanism: the application of passthrough mechanism to stack of self-attention blocks helps Transformer-like models get more fine-grained information with negligible extra computation costs. Our experiments on real-world datasets show that our TCCT architectures could greatly improve the performance of existing state-of-art Transformer models on time series forecasting with much lower computation and memory costs, including canonical Transformer, LogTrans and Informer.