# Self-Attentive Ensemble Transformer: Representing Ensemble Interactions in Neural Networks for Earth System Models

arXiv: https://arxiv.org/abs/2106.13924 · HTML: https://mlsanity.com/paper/2106.13924/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 42 references that are in the map.
- First posted on arXiv 21 Jun 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [Fixup Initialization: Residual Learning Without Normalization](https://mlsanity.com/paper/1901.09321/) (2019)
- [Neural networks for post-processing ensemble weather forecasts](https://mlsanity.com/paper/1805.09091/) (2018)
- [Machine learning methods for postprocessing ensemble forecasts of wind gusts: A systematic comparison](https://mlsanity.com/paper/2106.09512/) (2021)
- [Uncertainty Quantification in Complex Simulation Models Using Ensemble Copula Coupling](https://mlsanity.com/paper/1302.7149/) (2013)
- [Combining predictive distributions for statistical post-processing of ensemble forecasts](https://mlsanity.com/paper/1607.08096/) (2016)
- [Deep Learning for Climate Model Output Statistics](https://mlsanity.com/paper/2012.10394/) (2020)

## Built on by (most cited first, 7 of 7)

- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [SEEDS: Emulation of Weather Forecast Ensembles with Diffusion Models](https://mlsanity.com/paper/2306.14066/) (2023)
- [ENS-10: A Dataset For Post-Processing Ensemble Weather Forecasts](https://mlsanity.com/paper/2206.14786/) (2022)
- [Postprocessing of Ensemble Weather Forecasts Using Permutation-invariant Neural Networks](https://mlsanity.com/paper/2309.04452/) (2023)
- [Self-attentive Transformer for Fast and Accurate Postprocessing of Temperature and Wind Speed Forecasts](https://mlsanity.com/paper/2412.13957/) (2024)
- [Advancing Subseasonal Forecasting with Machine Learning](https://mlsanity.com/paper/2604.16238/) (2026)
- [Ensemble-size-dependence of deep-learning post-processing methods that minimize an (un)fair score: motivating examples and a proof-of-concept solution](https://mlsanity.com/paper/2602.15830/) (2026)

## Abstract (arXiv)

Ensemble data from Earth system models has to be calibrated and post-processed. I propose a novel member-by-member post-processing approach with neural networks. I bridge ideas from ensemble data assimilation with self-attention, resulting into the self-attentive ensemble transformer. Here, interactions between ensemble members are represented as additive and dynamic self-attentive part. As proof-of-concept, I regress global ECMWF ensemble forecasts to 2-metre-temperature fields from the ERA5 reanalysis. I demonstrate that the ensemble transformer can calibrate the ensemble spread and extract additional information from the ensemble. As it is a member-by-member approach, the ensemble transformer directly outputs multivariate and spatially-coherent ensemble members. Therefore, self-attention and the transformer technique can be a missing piece for a non-parametric post-processing of ensemble data with neural networks.