# Local Off-Grid Weather Forecasting with Multi-Modal Earth Observation Data

arXiv: https://arxiv.org/abs/2410.12938 · HTML: https://mlsanity.com/paper/2410.12938/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 52 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [Message Passing Neural PDE Solvers](https://mlsanity.com/paper/2202.03376/) (2022)
- [ClimaX: A foundation model for weather and climate](https://mlsanity.com/paper/2301.10343/) (2023)
- [Neural General Circulation Models for Weather and Climate](https://mlsanity.com/paper/2311.07222/) (2023)
- [DeepSD: Generating High Resolution Climate Change Projections through Single Image Super-Resolution](https://mlsanity.com/paper/1703.03126/) (2017)
- [Deep Learning for Day Forecasts from Sparse Observations](https://mlsanity.com/paper/2306.06079/) (2023)
- [AtmoRep: A stochastic model of atmosphere dynamics using large scale representation learning](https://mlsanity.com/paper/2308.13280/) (2023)
- [Hard-Constrained Deep Learning for Climate Downscaling](https://mlsanity.com/paper/2208.05424/) (2022)
- [Kilometer-Scale Convection Allowing Model Emulation using Generative Diffusion Modeling](https://mlsanity.com/paper/2408.10958/) (2024)
- [Prithvi WxC: Foundation Model for Weather and Climate](https://mlsanity.com/paper/2409.13598/) (2024)
- [MAgNet: Mesh Agnostic Neural PDE Solver](https://mlsanity.com/paper/2210.05495/) (2022)
- [Fourier Neural Operators for Arbitrary Resolution Climate Data Downscaling](https://mlsanity.com/paper/2305.14452/) (2023)
- [Graph-based Neural Weather Prediction for Limited Area Modeling](https://mlsanity.com/paper/2309.17370/) (2023)
- [Adaptive Bias Correction for Improved Subseasonal Forecasting](https://mlsanity.com/paper/2209.10666/) (2022)
- [Evaluating the transferability potential of deep learning models for climate downscaling](https://mlsanity.com/paper/2407.12517/) (2024)
- [Spatio-Temporal Wind Speed Forecasting using Graph Networks and Novel Transformer Architectures](https://mlsanity.com/paper/2208.13585/) (2022)
- [Verification against in-situ observations for Data-Driven Weather Prediction](https://mlsanity.com/paper/2305.00048/) (2023)

## Built on by (most cited first, 9 of 9)

- [TerraMind: Large-Scale Generative Multimodality for Earth Observation](https://mlsanity.com/paper/2504.11171/) (2025)
- [Kolmogorov Arnold Neural Interpolator for Downscaling and Correcting Meteorological Fields from In-Situ Observations](https://mlsanity.com/paper/2501.14404/) (2025)
- [Regional Weather Variable Predictions by Machine Learning with Near-Surface Observational and Atmospheric Numerical Data](https://mlsanity.com/paper/2412.10450/) (2024)
- [Conformal Prediction for Generative Models via Adaptive Cluster-Based Density Estimation](https://mlsanity.com/paper/2601.22298/) (2026)
- [Observation-driven correction of numerical weather prediction for marine winds](https://mlsanity.com/paper/2512.03606/) (2025)
- [Breaking the Regional Barrier: Inductive Semantic Topology Learning for Worldwide Air Quality Forecasting](https://mlsanity.com/paper/2601.21899/) (2026)
- [Partial recovery of meter-scale surface weather](https://mlsanity.com/paper/2602.23146/) (2026)
- [Accurate, Efficient, and Explainable Deep Learning Approaches for Environmental Science Problems](https://mlsanity.com/paper/2605.19366/) (2026)
- [Spatial Support Matters: Geometry-Aware Graph Fusion for Rainfall Field Reconstruction](https://mlsanity.com/paper/2607.01621/) (2026)

## Abstract (arXiv)

Urgent applications like wildfire management and renewable energy generation require precise, localized weather forecasts near the Earth's surface. However, forecasts produced by machine learning models or numerical weather prediction systems are typically generated on large-scale regular grids, where direct downscaling fails to capture fine-grained, near-surface weather patterns. In this work, we propose a multi-modal transformer model trained end-to-end to downscale gridded forecasts to off-grid locations of interest. Our model directly combines local historical weather observations (e.g., wind, temperature, dewpoint) with gridded forecasts to produce locally accurate predictions at various lead times. Multiple data modalities are collected and concatenated at station-level locations, treated as a token at each station. Using self-attention, the token corresponding to the target location aggregates information from its neighboring tokens. Experiments using weather stations across the Northeastern United States show that our model outperforms a range of data-driven and non-data-driven off-grid forecasting methods. They also reveal that direct input of station data provides a phase shift in local weather forecasting accuracy, reducing the prediction error by up to 80% compared to pure gridded data based models. This approach demonstrates how to bridge the gap between large-scale weather models and locally accurate forecasts to support high-stakes, location-sensitive decision-making.