# On the definition and importance of interpretability in scientific machine learning

arXiv: https://arxiv.org/abs/2505.13510 · HTML: https://mlsanity.com/paper/2505.13510/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 79 references that are in the map.
- First posted on arXiv 16 May 2025.

## Builds on (most cited first, 27 of 27)

- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [European Union regulations on algorithmic decision-making and a "right to explanation"](https://mlsanity.com/paper/1606.08813/) (2016)
- [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019)
- [Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1801.06637/) (2018)
- [Interpretable Scientific Discovery with Symbolic Regression: A Review](https://mlsanity.com/paper/2211.10873/) (2022)
- [The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches](https://mlsanity.com/paper/1803.01164/) (2018)
- [Rediscovering orbital mechanics with machine learning](https://mlsanity.com/paper/2202.02306/) (2022)
- [A Bayesian machine scientist to aid in the solution of challenging scientific problems](https://mlsanity.com/paper/2004.12157/) (2020)
- [Unsupervised discovery of interpretable hyperelastic constitutive laws](https://mlsanity.com/paper/2010.13496/) (2020)
- [LNO: Laplace Neural Operator for Solving Differential Equations](https://mlsanity.com/paper/2303.10528/) (2023)
- [Data-driven Tissue Mechanics with Polyconvex Neural Ordinary Differential Equations](https://mlsanity.com/paper/2110.03774/) (2021)
- [Discovering Sparse Interpretable Dynamics from Partial Observations](https://mlsanity.com/paper/2107.10879/) (2021)
- [SINDy-BVP: Sparse Identification of Nonlinear Dynamics for Boundary Value Problems](https://mlsanity.com/paper/2005.10756/) (2020)
- [Symbolic regression for scientific discovery: an application to wind speed forecasting](https://mlsanity.com/paper/2102.10570/) (2021)
- [Extreme sparsification of physics-augmented neural networks for interpretable model discovery in mechanics](https://mlsanity.com/paper/2310.03652/) (2023)
- [A toolkit for data-driven discovery of governing equations in high-noise regimes](https://mlsanity.com/paper/2111.04870/) (2021)
- [Neural Green's Operators for Parametric Partial Differential Equations](https://mlsanity.com/paper/2406.01857/) (2024)
- [Investigating the Duality of Interpretability and Explainability in Machine Learning](https://mlsanity.com/paper/2503.21356/) (2025)
- [GINN-LP: A Growing Interpretable Neural Network for Discovering Multivariate Laurent Polynomial Equations](https://mlsanity.com/paper/2312.10913/) (2023)
- [Discovering interpretable Lagrangian of dynamical systems from data](https://mlsanity.com/paper/2302.04400/) (2023)
- [Learning Adaptive Hydrodynamic Models Using Neural ODEs in Complex Conditions](https://mlsanity.com/paper/2410.00490/) (2024)
- [Discovering an interpretable mathematical expression for a full wind-turbine wake with artificial intelligence enhanced symbolic regression](https://mlsanity.com/paper/2406.00695/) (2024)
- [Interpretable and Efficient Data-driven Discovery and Control of Distributed Systems](https://mlsanity.com/paper/2411.04098/) (2024)

## Built on by (most cited first, 5 of 5)

- [Physics-informed sensor coverage through structure preserving machine learning](https://mlsanity.com/paper/2509.10363/) (2025)
- [Learning Gradient Flow: Using Equation Discovery to Accelerate Engineering Optimization](https://mlsanity.com/paper/2602.13513/) (2026)
- [Physically Interpretable Representation Learning with Gaussian Mixture Variational AutoEncoder (GM-VAE)](https://mlsanity.com/paper/2511.21883/) (2025)
- [Performance and Explainability Requirements of Evolutionary Algorithms in Real-World Physics-Informed Optimization](https://mlsanity.com/paper/2605.28164/) (2026)
- [The limits of interpretability in multiple linear regression](https://mlsanity.com/paper/2606.16013/) (2026)

## Abstract (arXiv)

Though neural networks trained on large datasets have been successfully used to describe and predict many physical phenomena, there is a sense among scientists that, unlike traditional scientific models comprising simple mathematical expressions, their findings cannot be integrated into the body of scientific knowledge. Critics of machine learning's inability to produce human-understandable relationships have converged on the concept of "interpretability" as its point of departure from more traditional forms of science. As the growing interest in interpretability has shown, researchers in the physical sciences seek not just predictive models, but also to uncover the fundamental principles that govern a system of interest. However, clarity around a definition of interpretability and the precise role that it plays in science is lacking in the literature. In this work, we argue that researchers in equation discovery and symbolic regression tend to conflate the concept of sparsity with interpretability. We review key papers on interpretable machine learning from outside the scientific community and argue that, though the definitions and methods they propose can inform questions of interpretability for scientific machine learning (SciML), they are inadequate for this new purpose. Noting these deficiencies, we propose an operational definition of interpretability for the physical sciences. Our notion of interpretability emphasizes understanding of the mechanism over mathematical sparsity. Innocuous though it may seem, this emphasis on mechanism shows that sparsity is often unnecessary. It also questions the possibility of interpretable scientific discovery when prior knowledge is lacking. We believe a precise and philosophically informed definition of interpretability in SciML will help focus research efforts toward the most significant obstacles to realizing a data-driven scientific future.