Interpreting "Interpretability" and Explaining "Explainability" in Machine Learning in Physics
Not on the map. Why?
Abstract (arXiv)
We review the concepts of interpretability and explainability as they apply to machine learning in physics. We define interpretability as concerning the structural transparency of a model (the ability to understand or approximate its inner workings) and explainability as concerning the scientific content of a model (the ability to map it onto domain knowledge). We discuss the trade-offs each entails (interpretability vs. expressivity; explainability vs. adaptability), the contexts in which each is needed, and the intrinsic and post-hoc tools available for achieving them. Throughout, we emphasize that machine-learned models are subject to the same scientific questions as classical models, differing only in scale, and that interpretability and explainability are best understood as deliberate modeling choices rather than inherent properties. We also emphasize the importance of task specification and intervention plans as a core aspect of model design.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 48 of its 112 references that are in the map. First posted on arXiv 24 Jun 2026.
Builds on
48 of 112 references.
References · 48
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
- 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
- 2016Understanding intermediate layers using linear classifier probescited by 1,353
- 2018On the Spectral Bias of Neural Networkscited by 1,341
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
- 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
36 more
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
- 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
- 2019Designing and Interpreting Probes with Control Taskscited by 640
- 2024Scaling and evaluating sparse autoencoderscited by 553
- 2019Benign Overfitting in Linear Regressioncited by 549
- 2019The frontier of simulation-based inferencecited by 502
- 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
- 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 368
- 2013k-Sparse Autoencoderscited by 215
- 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 191
- 2018Fast inference of deep neural networks in FPGAs for particle physicscited by 133
- 2018Discovering physical concepts with neural networkscited by 117
- 2019On the Existence of Simpler Machine Learning Modelscited by 89
- 2018Energy Flow Networks: Deep Sets for Particle Jetscited by 84
- 2015Rotation-invariant convolutional neural networks for galaxy morphology predictioncited by 83
- 2021A Living Review of Machine Learning for Particle Physicscited by 78
- 2024Lorentz-Equivariant Geometric Algebra Transformers for High-Energy Physicscited by 38
- 2024CaloChallenge 2022: A Community Challenge for Fast Calorimeter Simulationcited by 35
- 2025hls4ml: A Flexible, Open-Source Platform for Deep Learning Acceleration on Reconfigurable Hardwarecited by 9
- 2023E(2) Equivariant Neural Networks for Robust Galaxy Morphology Classificationcited by 7
- 2022Predicting the Thermal Sunyaev-Zel'dovich Field using Modular and Equivariant Set-Based Neural Networkscited by 6
- 2022Discovering the building blocks of dark matter halo density profiles with neural networkscited by 5
- 2022DeepLSS: breaking parameter degeneracies in large scale structure with deep learning analysis of combined probescited by 5
- 2022FAIR AI Models in High Energy Physicscited by 4
- 2023Physics-informed neural networks in the recreation of hydrodynamic simulations from dark mattercited by 4
- 2023Explaining dark matter halo density profiles with neural networkscited by 4
- 2020Deep learning insights into cosmological structure formationcited by 3
- 2023Efficient and Robust Jet Tagging at the LHC with Knowledge Distillationcited by 3
- 2026Explainable AI for Jet Tagging: A Comparative Study of GNNExplainer, GNNShap, and GradCAM for Jet Tagging in the Lund Jet Planecited by 3
- 2025The Pareto Frontier of Resilient Jet Taggingcited by 2
- 2026Variance reduction in lattice QCD observables via normalizing flowscited by 2
- 2022Insights into the origin of halo mass profiles from machine learningcited by 1
- 2025LambdaCDM and early dark energy in latent space: a data-driven parametrization of the CMB temperature power spectrumcited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.