# TENT: Tensorized Encoder Transformer for Temperature Forecasting

arXiv: https://arxiv.org/abs/2106.14742 · HTML: https://mlsanity.com/paper/2106.14742/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 52 references that are in the map.
- First posted on arXiv 28 Jun 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Generating Wikipedia by Summarizing Long Sequences](https://mlsanity.com/paper/1801.10198/) (2018)
- [Skillful Precipitation Nowcasting using Deep Generative Models of Radar](https://mlsanity.com/paper/2104.00954/) (2021)
- [A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction](https://mlsanity.com/paper/1704.02971/) (2017)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [Constituency Parsing with a Self-Attentive Encoder](https://mlsanity.com/paper/1805.01052/) (2018)
- [Temporal Pattern Attention for Multivariate Time Series Forecasting](https://mlsanity.com/paper/1809.04206/) (2018)
- [Star-Transformer](https://mlsanity.com/paper/1902.09113/) (2019)
- [A Tensorized Transformer for Language Modeling](https://mlsanity.com/paper/1906.09777/) (2019)
- [DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding](https://mlsanity.com/paper/1709.04696/) (2017)
- [SmaAt-UNet: Precipitation Nowcasting using a Small Attention-UNet Architecture](https://mlsanity.com/paper/2007.04417/) (2020)
- [A Comparison of LSTM and BERT for Small Corpus](https://mlsanity.com/paper/2009.05451/) (2020)
- [Broad-UNet: Multi-scale feature learning for nowcasting tasks](https://mlsanity.com/paper/2102.06442/) (2021)
- [AFS: An Attention-based mechanism for Supervised Feature Selection](https://mlsanity.com/paper/1902.11074/) (2019)
- [Deep Graph Convolutional Networks for Wind Speed Prediction](https://mlsanity.com/paper/2101.10041/) (2021)
- [Wind speed prediction using multidimensional convolutional neural networks](https://mlsanity.com/paper/2007.12567/) (2020)
- [Symbolic regression for scientific discovery: an application to wind speed forecasting](https://mlsanity.com/paper/2102.10570/) (2021)
- [Deep coastal sea elements forecasting using U-Net based models](https://mlsanity.com/paper/2011.03303/) (2020)
- [Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together](https://mlsanity.com/paper/1805.00912/) (2018)
- [BT-Nets: Simplifying Deep Neural Networks via Block Term Decomposition](https://mlsanity.com/paper/1712.05689/) (2017)

## Built on by (most cited first, 7 of 7)

- [Foundation Models for Weather and Climate Data Understanding: A Comprehensive Survey](https://mlsanity.com/paper/2312.03014/) (2023)
- [AA-TransUNet: Attention Augmented TransUNet For Nowcasting Tasks](https://mlsanity.com/paper/2202.04996/) (2022)
- [Multistream Graph Attention Networks for Wind Speed Forecasting](https://mlsanity.com/paper/2108.07063/) (2021)
- [Factor Augmented Tensor-on-Tensor Neural Networks](https://mlsanity.com/paper/2405.19610/) (2024)
- [WF-UNet: Weather Fusion UNet for Precipitation Nowcasting](https://mlsanity.com/paper/2302.04102/) (2023)
- [Interpolation of mountain weather forecasts by machine learning](https://mlsanity.com/paper/2308.13983/) (2023)
- [Integrating Weather Station Data and Radar for Precipitation Nowcasting: SmaAt-fUsion and SmaAt-Krige-GNet](https://mlsanity.com/paper/2502.16116/) (2025)

## Abstract (arXiv)

Reliable weather forecasting is of great importance in science, business, and society. The best performing data-driven models for weather prediction tasks rely on recurrent or convolutional neural networks, where some of which incorporate attention mechanisms. In this work, we introduce a novel model based on Transformer architecture for weather forecasting. The proposed Tensorial Encoder Transformer (TENT) model is equipped with tensorial attention and thus it exploits the spatiotemporal structure of weather data by processing it in multidimensional tensorial format. We show that compared to the classical encoder transformer, 3D convolutional neural networks, LSTM, and Convolutional LSTM, the proposed TENT model can better learn the underlying complex pattern of the weather data for the studied temperature prediction task. Experiments on two real-life weather datasets are performed. The datasets consist of historical measurements from weather stations in the USA, Canada and Europe. The first dataset contains hourly measurements of weather attributes for 30 cities in the USA and Canada from October 2012 to November 2017. The second dataset contains daily measurements of weather attributes of 18 cities across Europe from May 2005 to April 2020. Two attention scores are introduced based on the obtained tonsorial attention and are visualized in order to shed light on the decision-making process of our model and provide insight knowledge on the most important cities for the target cities.