# Probing the limit of hydrologic predictability with the Transformer network

arXiv: https://arxiv.org/abs/2306.12384 · HTML: https://mlsanity.com/paper/2306.12384/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 21 Jun 2023.

## Builds on (most cited first, 11 of 11)

- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Towards Learning Universal, Regional, and Local Hydrological Behaviors via Machine-Learning Applied to Large-Sample Datasets](https://mlsanity.com/paper/1907.08456/) (2019)
- [Enhancing streamflow forecast and extracting insights using long-short term memory networks with data integration at continental scales](https://mlsanity.com/paper/1912.08949/) (2019)
- [From calibration to parameter learning: Harnessing the scaling effects of big data in geoscientific modeling](https://mlsanity.com/paper/2007.15751/) (2020)
- [Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps](https://mlsanity.com/paper/2207.03380/) (2022)
- [Differentiable, learnable, regionalized process-based models with physical outputs can approach state-of-the-art hydrologic prediction accuracy](https://mlsanity.com/paper/2203.14827/) (2022)
- [The data synergy effects of time-series deep learning models in hydrology](https://mlsanity.com/paper/2101.01876/) (2021)
- [Prediction in ungauged regions with sparse flow duration curves and input-selection ensemble modeling](https://mlsanity.com/paper/2011.13380/) (2020)
- [Temporal Fusion Transformers for Streamflow Prediction: Value of Combining Attention with Recurrence](https://mlsanity.com/paper/2305.12335/) (2023)

## Built on by (most cited first, 9 of 9)

- [Time Series Predictions in Unmonitored Sites: A Survey of Machine Learning Techniques in Water Resources](https://mlsanity.com/paper/2308.09766/) (2023)
- [X-MethaneWet: A Cross-scale Global Wetland Methane Emission Benchmark Dataset for Advancing Science Discovery with AI](https://mlsanity.com/paper/2505.18355/) (2025)
- [HydroDiffusion: Diffusion-Based Probabilistic Streamflow Forecasting with a State Space Backbone](https://mlsanity.com/paper/2512.12183/) (2025)
- [StefaLand: An Efficient Geoscience Foundation Model That Improves Dynamic Land-Surface Predictions](https://mlsanity.com/paper/2509.17942/) (2025)
- [Learning Generative Models for Lumped Rainfall-Runoff Modeling](https://mlsanity.com/paper/2309.09904/) (2023)
- [Distinct hydrologic response patterns and trends worldwide revealed by physics-embedded learning](https://mlsanity.com/paper/2504.10707/) (2025)
- [Efficacy of Temporal Fusion Transformers for Runoff Simulation](https://mlsanity.com/paper/2506.20831/) (2025)
- [Rethinking deep learning: linear regression remains a key benchmark in predicting terrestrial water storage](https://mlsanity.com/paper/2510.10799/) (2025)
- [Context-Aware Concept Distillation for Trustworthy Flood Prediction](https://mlsanity.com/paper/2607.23237/) (2026)

## Abstract (arXiv)

For a number of years since its introduction to hydrology, recurrent neural networks like long short-term memory (LSTM) have proven remarkably difficult to surpass in terms of daily hydrograph metrics on known, comparable benchmarks. Outside of hydrology, Transformers have now become the model of choice for sequential prediction tasks, making it a curious architecture to investigate. Here, we first show that a vanilla Transformer architecture is not competitive against LSTM on the widely benchmarked CAMELS dataset, and lagged especially for the high-flow metrics due to short-term processes. However, a recurrence-free variant of Transformer can obtain mixed comparisons with LSTM, producing the same Kling-Gupta efficiency coefficient (KGE), along with other metrics. The lack of advantages for the Transformer is linked to the Markovian nature of the hydrologic prediction problem. Similar to LSTM, the Transformer can also merge multiple forcing dataset to improve model performance. While the Transformer results are not higher than current state-of-the-art, we still learned some valuable lessons: (1) the vanilla Transformer architecture is not suitable for hydrologic modeling; (2) the proposed recurrence-free modification can improve Transformer performance so future work can continue to test more of such modifications; and (3) the prediction limits on the dataset should be close to the current state-of-the-art model. As a non-recurrent model, the Transformer may bear scale advantages for learning from bigger datasets and storing knowledge. This work serves as a reference point for future modifications of the model.