# Towards mechanistic understanding in a data-driven weather model: internal activations reveal interpretable physical features

arXiv: https://arxiv.org/abs/2512.24440 · HTML: https://mlsanity.com/paper/2512.24440/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 40 references that are in the map.
- First posted on arXiv 30 Dec 2025.

## Builds on (most cited first, 17 of 17)

- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Neural General Circulation Models for Weather and Climate](https://mlsanity.com/paper/2311.07222/) (2023)
- [Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability](https://mlsanity.com/paper/2301.04709/) (2023)
- [Transcoders Find Interpretable LLM Feature Circuits](https://mlsanity.com/paper/2406.11944/) (2024)
- [Pangu-Weather: A 3D High-Resolution Model for Fast and Accurate Global Weather Forecast](https://mlsanity.com/paper/2211.02556/) (2022)
- [The Hydra Effect: Emergent Self-repair in Language Model Computations](https://mlsanity.com/paper/2307.15771/) (2023)
- [InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2412.12101/) (2024)
- [Skillful joint probabilistic weather forecasting from marginals](https://mlsanity.com/paper/2506.10772/) (2025)
- [Weight-sparse transformers have interpretable circuits](https://mlsanity.com/paper/2511.13653/) (2025)
- [Can AI weather models predict out-of-distribution gray swan tropical cyclones?](https://mlsanity.com/paper/2410.14932/) (2024)
- [Numerical models outperform AI weather forecasts of record-breaking extremes](https://mlsanity.com/paper/2508.15724/) (2025)
- [Panda: A pretrained forecast model for chaotic dynamics](https://mlsanity.com/paper/2505.13755/) (2025)

## Built on by (most cited first, 12 of 12)

- [Sparse probes and murky physics: a case study of interpretability challenges in a foundation model for continuum dynamics](https://mlsanity.com/paper/2606.11657/) (2026)
- [Mechanistic Interpretability Tool for AI Weather Models](https://mlsanity.com/paper/2604.20467/) (2026)
- [The physics of AI weather models](https://mlsanity.com/paper/2605.23778/) (2026)
- [Skillful Global Ocean Emulation and the Role of Correlation-Aware Loss](https://mlsanity.com/paper/2604.18727/) (2026)
- [Mechanistic Interpretability and Causal Feature Steering of Neural Quantum States via Sparse Autoencoders](https://mlsanity.com/paper/2607.01336/) (2026)
- [Toward Mechanistic Interpretability of an AI Foundation Model Fine-Tuned for Atmospheric Chemistry](https://mlsanity.com/paper/2607.20778/) (2026)
- [Beyond Linear Superposition: Discovering Climate Features in AI Weather Models with KAN-SAE](https://mlsanity.com/paper/2605.17493/) (2026)
- [No Epoch Like the Present: Robust Climate Emulation Requires Out-of-Distribution Generalisation](https://mlsanity.com/paper/2605.22248/) (2026)
- [Does Aurora Encode Atmospheric Structure? Latent Regime Analysis and Attribution](https://mlsanity.com/paper/2606.26361/) (2026)
- [S2S-JEPA: Predicting the Predictable at Subseasonal-to-Seasonal Timescales](https://mlsanity.com/paper/2610.03106/) (2026)
- [Mechanistic Interpretability of Atmospheric Rivers in GraphCast](https://mlsanity.com/paper/2610.07583/) (2026)
- [Strategic Governance of AI Models in Earth Science](https://mlsanity.com/paper/2610.10560/) (2026)

## Abstract (arXiv)

Large data-driven physics models like DeepMind's weather model GraphCast have empirically succeeded in parameterizing time operators for complex dynamical systems with an accuracy reaching or in some cases exceeding that of traditional physics-based solvers. Unfortunately, how these data-driven models perform computations is largely unknown and whether their internal representations are interpretable or physically consistent is an open question. Here, we adapt tools from interpretability research in Large Language Models to analyze intermediate computational layers in GraphCast, leveraging sparse autoencoders to discover interpretable features in the neuron space of the model. We uncover distinct features on a wide range of length and time scales that correspond to tropical cyclones, atmospheric rivers, diurnal and seasonal behavior, large-scale precipitation patterns, specific geographical coding, and sea-ice extent, among others. We further demonstrate how the precise abstraction of these features can be probed via interventions on the prediction steps of the model. As a case study, we sparsely modify a feature corresponding to tropical cyclones in GraphCast and observe interpretable and physically consistent modifications to evolving hurricanes. Such methods offer a window into the black-box behavior of data-driven physics models and are a step towards realizing their potential as trustworthy predictors and scientifically valuable tools for discovery.