# ClimaX: A foundation model for weather and climate

arXiv: https://arxiv.org/abs/2301.10343 · HTML: https://mlsanity.com/paper/2301.10343/

## Key facts

- Cited by 337 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 127 references that are in the map.
- First posted on arXiv 24 Jan 2023.

## Builds on (most cited first, 30 of 58)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)

## Built on by (most cited first, 30 of 337)

- [DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models](https://mlsanity.com/paper/2309.14509/) (2023)
- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers](https://mlsanity.com/paper/2308.05732/) (2023)
- [Poseidon: Efficient Foundation Models for PDEs](https://mlsanity.com/paper/2405.19101/) (2024)
- [DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2403.03542/) (2024)
- [Foundation Models for Generalist Geospatial Artificial Intelligence](https://mlsanity.com/paper/2310.18660/) (2023)
- [Scalable Transformer for PDE Surrogate Modeling](https://mlsanity.com/paper/2305.17560/) (2023)
- [An Expert's Guide to Training Physics-informed Neural Networks](https://mlsanity.com/paper/2308.08468/) (2023)
- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](https://mlsanity.com/paper/2310.10196/) (2023)
- [Scaling transformer neural networks for skillful and reliable medium-range weather forecasting](https://mlsanity.com/paper/2312.03876/) (2023)
- [DYffusion: A Dynamics-informed Diffusion Model for Spatiotemporal Forecasting](https://mlsanity.com/paper/2306.01984/) (2023)
- [Multiple Physics Pretraining for Physical Surrogate Models](https://mlsanity.com/paper/2310.02994/) (2023)
- [Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data](https://mlsanity.com/paper/2402.05892/) (2024)
- [ClimODE: Climate and Weather Forecasting with Physics-informed Neural ODEs](https://mlsanity.com/paper/2404.10024/) (2024)
- [Deep Learning for Day Forecasts from Sparse Observations](https://mlsanity.com/paper/2306.06079/) (2023)
- [Large AI Models in Health Informatics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2303.11568/) (2023)
- [ACE: A fast, skillful learned global atmospheric model for climate prediction](https://mlsanity.com/paper/2310.02074/) (2023)
- [CasCast: Skillful High-resolution Precipitation Nowcasting via Cascaded Modelling](https://mlsanity.com/paper/2402.04290/) (2024)
- [AB-UPT: Scaling Neural CFD Surrogates for High-Fidelity Automotive Aerodynamics Simulations via Anchored-Branched Universal Physics Transformers](https://mlsanity.com/paper/2502.09692/) (2025)
- [Vision-LSTM: xLSTM as Generic Vision Backbone](https://mlsanity.com/paper/2406.04303/) (2024)
- [ClimateLearn: Benchmarking Machine Learning for Weather and Climate Modeling](https://mlsanity.com/paper/2307.01909/) (2023)
- [Kolmogorov-Arnold Network for Satellite Image Classification in Remote Sensing](https://mlsanity.com/paper/2406.00600/) (2024)
- [ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction](https://mlsanity.com/paper/2402.00712/) (2024)
- [AtmoRep: A stochastic model of atmosphere dynamics using large scale representation learning](https://mlsanity.com/paper/2308.13280/) (2023)
- [Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRs](https://mlsanity.com/paper/2412.16178/) (2024)
- [Geometric Clifford Algebra Networks](https://mlsanity.com/paper/2302.06594/) (2023)
- [Walrus: A Cross-Domain Foundation Model for Continuum Dynamics](https://mlsanity.com/paper/2511.15684/) (2025)
- [Visual Tuning](https://mlsanity.com/paper/2305.06061/) (2023)
- [Foundation Models for Weather and Climate Data Understanding: A Comprehensive Survey](https://mlsanity.com/paper/2312.03014/) (2023)
- [Panopticon: Advancing Any-Sensor Foundation Models for Earth Observation](https://mlsanity.com/paper/2503.10845/) (2025)

## Abstract (arXiv)

Most state-of-the-art approaches for weather and climate modeling are based on physics-informed numerical models of the atmosphere. These approaches aim to model the non-linear dynamics and complex interactions between multiple variables, which are challenging to approximate. Additionally, many such numerical models are computationally intensive, especially when modeling the atmospheric phenomenon at a fine-grained spatial and temporal resolution. Recent data-driven approaches based on machine learning instead aim to directly solve a downstream forecasting or projection task by learning a data-driven functional mapping using deep neural networks. However, these networks are trained using curated and homogeneous climate datasets for specific spatiotemporal tasks, and thus lack the generality of numerical models. We develop and demonstrate ClimaX, a flexible and generalizable deep learning model for weather and climate science that can be trained using heterogeneous datasets spanning different variables, spatio-temporal coverage, and physical groundings. ClimaX extends the Transformer architecture with novel encoding and aggregation blocks that allow effective use of available compute while maintaining general utility. ClimaX is pre-trained with a self-supervised learning objective on climate datasets derived from CMIP6. The pre-trained ClimaX can then be fine-tuned to address a breadth of climate and weather tasks, including those that involve atmospheric variables and spatio-temporal scales unseen during pretraining. Compared to existing data-driven baselines, we show that this generality in ClimaX results in superior performance on benchmarks for weather forecasting and climate projections, even when pretrained at lower resolutions and compute budgets. The source code is available at https://github.com/microsoft/ClimaX.