# Sparse probes and murky physics: a case study of interpretability challenges in a foundation model for continuum dynamics

arXiv: https://arxiv.org/abs/2606.11657 · HTML: https://mlsanity.com/paper/2606.11657/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability.
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 10 Jun 2026.

## Placed in

- Mechanistic interpretability / [Sparse autoencoder training and evaluation](https://mlsanity.com/mechinterp/sparse-autoencoder-training-and-evaluation/)

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [Poseidon: Efficient Foundation Models for PDEs](https://mlsanity.com/paper/2405.19101/) (2024)
- [The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning](https://mlsanity.com/paper/2412.00568/) (2024)
- [Multiple Physics Pretraining for Physical Surrogate Models](https://mlsanity.com/paper/2310.02994/) (2023)
- [Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics](https://mlsanity.com/paper/2410.21272/) (2024)
- [A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models](https://mlsanity.com/paper/2503.05613/) (2025)
- [Walrus: A Cross-Domain Foundation Model for Continuum Dynamics](https://mlsanity.com/paper/2511.15684/) (2025)
- [An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation](https://mlsanity.com/paper/2410.03334/) (2024)
- [Interpretable Machine Learning in Physics: A Review](https://mlsanity.com/paper/2503.23616/) (2025)
- [Towards mechanistic understanding in a data-driven weather model: internal activations reveal interpretable physical features](https://mlsanity.com/paper/2512.24440/) (2025)
- [Southern Ocean Dynamics Under Climate Change: New Knowledge Through Physics-Guided Machine Learning](https://mlsanity.com/paper/2310.13916/) (2023)
- [Overtone: Cyclic Patch Modulation for Clean, Efficient, and Flexible Physics Emulators](https://mlsanity.com/paper/2507.09264/) (2025)
- [Physics Steering: Causal Control of Cross-Domain Concepts in a Physics Foundation Model](https://mlsanity.com/paper/2511.20798/) (2025)
- [OceanCBM: A Concept Bottleneck Model for Mechanistic Interpretability in Ocean Forecasting](https://mlsanity.com/paper/2605.12639/) (2026)

## Built on by (most cited first, 3 of 3)

- [Mechanistic Interpretability and Causal Feature Steering of Neural Quantum States via Sparse Autoencoders](https://mlsanity.com/paper/2607.01336/) (2026)
- [Feature Superposition in Neural Networks: From Theory to Practice](https://mlsanity.com/paper/2609.06862/) (2026)
- [A survey detection channel overrides the pixels in an astronomical foundation model, and biases tomographic mean redshifts](https://mlsanity.com/paper/2608.23626/) (2026)

## Abstract (arXiv)

Generative AI emulators are increasingly used in scientific domains where we already have strong theory, benchmarks, and physical intuition. This raises a central evaluation and interpretability question: when a foundation-style model can reproduce known continuum dynamics, what internal mechanism supports that behavior, is the internal behaviour consistent with known physics, and how does it relate to where the emulator succeeds or fails? We investigate a cross-domain foundation model for continuum dynamics, Walrus by Polymathic, using mechanistic interpretability guided by physical principles. We apply a sparse autoencoder (SAE) to probe a selected layer, and address the practical challenge of triaging a large feature set (over 20,000) using enstrophy as a physically grounded metric. As a deliberately simple testbed, we focus on shear flow and compare feature recruitment across multiple shear-flow setups, i.e. parameter values in the numerical simulation. Across setups we find evidence of piecewise consistency, with subsets of features recurring in similar roles, but this structure is intermittent and does not map cleanly onto standard physical decompositions. In parallel, direct comparisons between numerical simulation and the emulator reveal systematic output-level discrepancies, including regimes where energy/structures become too diffuse or too localized. We connect parts of these discrepancies to changes in specific SAE feature usage. Our work highlights open questions for scientific foundation models: how to robustly prioritize mechanistically meaningful features, how to separate stable structure from analysis artifacts (including single-layer and SAE limitations), and how to use established benchmarks to decide when "different" internal representations are genuinely informative rather than merely effective.