# Temporal Fusion Transformers for Streamflow Prediction: Value of Combining Attention with Recurrence

arXiv: https://arxiv.org/abs/2305.12335 · HTML: https://mlsanity.com/paper/2305.12335/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 44 references that are in the map.
- First posted on arXiv 21 May 2023.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Towards Learning Universal, Regional, and Local Hydrological Behaviors via Machine-Learning Applied to Large-Sample Datasets](https://mlsanity.com/paper/1907.08456/) (2019)
- [Enhancing streamflow forecast and extracting insights using long-short term memory networks with data integration at continental scales](https://mlsanity.com/paper/1912.08949/) (2019)
- [Ranger21: a synergistic deep learning optimizer](https://mlsanity.com/paper/2106.13731/) (2021)
- [Prolongation of SMAP to Spatio-temporally Seamless Coverage of Continental US Using a Deep Learning Neural Network](https://mlsanity.com/paper/1707.06611/) (2017)

## Built on by (most cited first, 4 of 4)

- [Probing the limit of hydrologic predictability with the Transformer network](https://mlsanity.com/paper/2306.12384/) (2023)
- [Using Machine Learning to Discover Parsimonious and Physically-Interpretable Representations of Catchment-Scale Rainfall-Runoff Dynamics](https://mlsanity.com/paper/2412.04845/) (2024)
- [Efficacy of Temporal Fusion Transformers for Runoff Simulation](https://mlsanity.com/paper/2506.20831/) (2025)
- [Rethinking deep learning: linear regression remains a key benchmark in predicting terrestrial water storage](https://mlsanity.com/paper/2510.10799/) (2025)

## Abstract (arXiv)

Over the past few decades, the hydrology community has witnessed notable advancements in streamflow prediction, particularly with the introduction of cutting-edge machine-learning algorithms. Recurrent neural networks, especially Long Short-Term Memory (LSTM) networks, have become popular due to their capacity to create precise forecasts and realistically mimic the system dynamics. Attention-based models, such as Transformers, can learn from the entire data sequence concurrently, a feature that LSTM does not have. This work tests the hypothesis that combining recurrence with attention can improve streamflow prediction. We set up the Temporal Fusion Transformer (TFT) architecture, a model that combines both of these aspects and has never been applied in hydrology before. We compare the performance of LSTM, Transformers, and TFT over 2,610 globally distributed catchments from the recently available Caravan dataset. Our results demonstrate that TFT indeed exceeds the performance benchmark set by the LSTM and Transformers for streamflow prediction. Additionally, being an explainable AI method, TFT helps in gaining insights into the streamflow generation processes.