# HEAL-ViT: Vision Transformers on a spherical mesh for medium-range weather forecasting

arXiv: https://arxiv.org/abs/2403.17016 · HTML: https://mlsanity.com/paper/2403.17016/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 14 Feb 2024.

## Builds on (most cited first, 13 of 13)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [FuXi: A cascade machine learning forecasting system for 15-day global weather forecast](https://mlsanity.com/paper/2306.12873/) (2023)
- [FractalNet: Ultra-Deep Neural Networks without Residuals](https://mlsanity.com/paper/1605.07648/) (2016)
- [DeepSphere: Efficient spherical Convolutional Neural Network with HEALPix sampling for cosmological applications](https://mlsanity.com/paper/1810.12186/) (2018)
- [SwinVRNN: A Data-Driven Ensemble Forecasting Model via Learned Distribution Perturbation](https://mlsanity.com/paper/2205.13158/) (2022)
- [SwinRDM: Integrate SwinRNN with Diffusion Model towards High-Resolution and High-Quality Weather Forecasting](https://mlsanity.com/paper/2306.03110/) (2023)

## Built on by (most cited first, 8 of 8)

- [Huge Ensembles Part I: Design of Ensemble Weather Forecasts using Spherical Fourier Neural Operators](https://mlsanity.com/paper/2408.03100/) (2024)
- [Deep Learning and Foundation Models for Weather Prediction: A Survey](https://mlsanity.com/paper/2501.06907/) (2025)
- [PEAR: Equal Area Weather Forecasting on the Sphere](https://mlsanity.com/paper/2505.17720/) (2025)
- [(Sparse) Attention to the Details: Preserving Spectral Fidelity in ML-based Weather Forecasting Models](https://mlsanity.com/paper/2604.16429/) (2026)
- [VarteX: Enhancing Weather Forecast through Distributed Variable Representation](https://mlsanity.com/paper/2406.19615/) (2024)
- [Eddy-Resolving Global Ocean Forecasting with Multi-Scale Graph Neural Networks](https://mlsanity.com/paper/2601.12775/) (2026)
- [Accurate, Efficient, and Explainable Deep Learning Approaches for Environmental Science Problems](https://mlsanity.com/paper/2605.19366/) (2026)
- [S3N: A Spherical Spiral Scanning Network for Weather Forecasting](https://mlsanity.com/paper/2610.04338/) (2026)

## Abstract (arXiv)

In recent years, a variety of ML architectures and techniques have seen success in producing skillful medium range weather forecasts. In particular, Vision Transformer (ViT)-based models (e.g. Pangu-Weather, FuXi) have shown strong performance, working nearly "out-of-the-box" by treating weather data as a multi-channel image on a rectilinear grid. While a rectilinear grid is appropriate for 2D images, weather data is inherently spherical and thus heavily distorted at the poles on a rectilinear grid, leading to disproportionate compute being used to model data near the poles. Graph-based methods (e.g. GraphCast) do not suffer from this problem, as they map the longitude-latitude grid to a spherical mesh, but are generally more memory intensive and tend to need more compute resources for training and inference. While spatially homogeneous, the spherical mesh does not lend itself readily to be modeled by ViT-based models that implicitly rely on the rectilinear grid structure. We present HEAL-ViT, a novel architecture that uses ViT models on a spherical mesh, thus benefiting from both the spatial homogeneity enjoyed by graph-based models and efficient attention-based mechanisms exploited by transformers. HEAL-ViT produces weather forecasts that outperform the ECMWF IFS on key metrics, and demonstrate better bias accumulation and blurring than other ML weather prediction models. Further, the lowered compute footprint of HEAL-ViT makes it attractive for operational use as well, where other models in addition to a 6-hourly prediction model may be needed to produce the full set of operational forecasts required.