# Mechanistic Interpretability Tool for AI Weather Models

arXiv: https://arxiv.org/abs/2604.20467 · HTML: https://mlsanity.com/paper/2604.20467/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Interpreting time-series and weather models, one of 13 directions in Mechanistic interpretability.
- Builds on 4 of its 17 references that are in the map.
- First posted on arXiv 22 Apr 2026.

## Placed in

- Mechanistic interpretability / [Interpreting time-series and weather models](https://mlsanity.com/mechinterp/interpreting-time-series-and-weather-models/)

## Builds on (most cited first, 4 of 4)

- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [Interpreting Attention Layer Outputs with Sparse Autoencoders](https://mlsanity.com/paper/2406.17759/) (2024)
- [Towards Uncovering How Large Language Model Works: An Explainability Perspective](https://mlsanity.com/paper/2402.10688/) (2024)
- [Towards mechanistic understanding in a data-driven weather model: internal activations reveal interpretable physical features](https://mlsanity.com/paper/2512.24440/) (2025)

## Built on by (most cited first, 2 of 2)

- [The physics of AI weather models](https://mlsanity.com/paper/2605.23778/) (2026)
- [Toward a Scientific Discovery Engine for Weather and Climate Data: A Visual Analytics Workbench for Embedding-Based Exploration](https://mlsanity.com/paper/2605.00972/) (2026)

## Abstract (arXiv)

Artificial Intelligence (AI) weather models are improving rapidly, and their forecasts are already competitive with long-established traditional Numerical Weather Prediction (NWP). To build confidence in this new methodology, it is critical that we understand how these predictions are generated. This is a huge challenge as these AI weather models remain largely black boxes. In other areas of Machine Learning (ML), mechanistic interpretability has emerged as a framework for understanding ML predictions by analysing the building blocks responsible for them. Here we present an open-source, highly adaptable tool which incorporates concepts from mechanistic interpretability. The tool organises internal latent representations from the model processor and allows for initial analyses, including cosine similarity and Principal Component Analysis (PCA), enabling the user to identify directions in latent space potentially associated with meteorological features. Applying our tool to the graph neural network GraphCast, we present preliminary case studies for mid-latitude synoptic-scale waves and specific humidity. These demonstrate the tool's ability to identify linear combinations of latent channels that appear to correspond to interpretable features.