# Probabilistic Transformers for Joint Modeling of Global Weather Dynamics and Decision-Centric Variables

arXiv: https://arxiv.org/abs/2601.03753 · HTML: https://mlsanity.com/paper/2601.03753/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 7 Jan 2026.

## Builds on (most cited first, 20 of 20)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [ClimaX: A foundation model for weather and climate](https://mlsanity.com/paper/2301.10343/) (2023)
- [FuXi: A cascade machine learning forecasting system for 15-day global weather forecast](https://mlsanity.com/paper/2306.12873/) (2023)
- [Neighborhood Attention Transformer](https://mlsanity.com/paper/2204.07143/) (2022)
- [Neural General Circulation Models for Weather and Climate](https://mlsanity.com/paper/2311.07222/) (2023)
- [GenCast: Diffusion-based ensemble forecasting for medium-range weather](https://mlsanity.com/paper/2312.15796/) (2023)
- [FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead](https://mlsanity.com/paper/2304.02948/) (2023)
- [Forecasting Global Weather with Graph Neural Networks](https://mlsanity.com/paper/2202.07575/) (2022)
- [Sub-seasonal forecasting with a large ensemble of deep-learning weather prediction models](https://mlsanity.com/paper/2102.05107/) (2021)
- [Skillful joint probabilistic weather forecasting from marginals](https://mlsanity.com/paper/2506.10772/) (2025)
- [FourCastNet 3: A geometric approach to probabilistic machine-learning weather forecasting at scale](https://mlsanity.com/paper/2507.12144/) (2025)
- [ArchesWeather & ArchesWeatherGen: a deterministic and generative model for efficient ML weather forecasting](https://mlsanity.com/paper/2412.12971/) (2024)
- [GraphDOP: Towards skilful data-driven medium-range weather forecasts learnt and initialised directly from observations](https://mlsanity.com/paper/2412.15687/) (2024)
- [FuXi Weather: A data-to-forecast machine learning system for global weather](https://mlsanity.com/paper/2408.05472/) (2024)
- [Analyzing and Exploring Training Recipes for Large-Scale Transformer-Based Weather Prediction](https://mlsanity.com/paper/2404.19630/) (2024)
- [FuXi-ENS: A machine learning model for medium-range ensemble weather forecasting](https://mlsanity.com/paper/2405.05925/) (2024)
- [Swift: An Autoregressive Consistency Model for Efficient Weather Forecasting](https://mlsanity.com/paper/2509.25631/) (2025)
- [OmniCast: A Masked Latent Diffusion Model for Weather Forecasting Across Time Scales](https://mlsanity.com/paper/2510.18707/) (2025)
- [Forecast Skill Is Not Decision Skill: Evidence from Weather-Dependent Decision Tasks](https://mlsanity.com/paper/2512.14779/) (2025)

## Built on by (most cited first, 1 of 1)

- [Timestep-Conditioned Transformers for Global Weather Forecasting](https://mlsanity.com/paper/2608.06241/) (2026)

## Abstract (arXiv)

Weather forecasts sit upstream of high-stakes decisions in domains such as grid operations, aviation, agriculture, and emergency response. Yet forecast users often face a difficult trade-off. Many decision-relevant targets are functionals of the atmospheric state variables, such as extrema, accumulations, and threshold exceedances, rather than state variables themselves. As a result, users must estimate these targets via post-processing, which can be suboptimal and can introduce structural bias. The core issue is that decisions depend on distributions over these functionals that the model is not trained to learn directly. In this work, we introduce GEM-2, a probabilistic transformer that jointly learns global atmospheric dynamics alongside a suite of variables that users directly act upon. Using this training recipe, we show that a lightweight (~275M params) and computationally efficient (~20-100x training speedup relative to state-of-the-art) transformer trained on the CRPS objective can directly outperform operational numerical weather prediction (NWP) models and be competitive with ML models that rely on expensive multi-step diffusion processes or require bespoke multi-stage fine-tuning strategies. We further demonstrate state-of-the-art economic value metrics under decision-theoretic evaluation, stable convergence to climatology at S2S and seasonal timescales, and a surprising insensitivity to many commonly assumed architectural and training design choices.