# Explain the Black Box for the Sake of Science: the Scientific Method in the Era of Generative Artificial Intelligence

arXiv: https://arxiv.org/abs/2406.10557 · HTML: https://mlsanity.com/paper/2406.10557/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 66 references that are in the map.
- First posted on arXiv 15 Jun 2024.

## Builds on (most cited first, 20 of 20)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Neural Additive Models: Interpretable Machine Learning with Neural Nets](https://mlsanity.com/paper/2004.13912/) (2020)
- [Explanations can be manipulated and geometry is to blame](https://mlsanity.com/paper/1906.07983/) (2019)
- [Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model](https://mlsanity.com/paper/1511.01644/) (2015)
- [How Transformers Learn Causal Structure with Gradient Descent](https://mlsanity.com/paper/2402.14735/) (2024)
- [Large Language Models for Automated Open-domain Scientific Hypotheses Discovery](https://mlsanity.com/paper/2309.02726/) (2023)
- [Unifying machine learning and quantum chemistry -- a deep neural network for molecular wavefunctions](https://mlsanity.com/paper/1906.10033/) (2019)
- [On scientific understanding with artificial intelligence](https://mlsanity.com/paper/2204.01467/) (2022)
- [Evaluation of post-hoc interpretability methods in time-series classification](https://mlsanity.com/paper/2202.05656/) (2022)
- [Machine Learning in the Search for New Fundamental Physics](https://mlsanity.com/paper/2112.03769/) (2021)
- [Interpretable Classification Models for Recidivism Prediction](https://mlsanity.com/paper/1503.07810/) (2015)
- [Causal Interpretation of Self-Attention in Pre-Trained Transformers](https://mlsanity.com/paper/2310.20307/) (2023)
- [Causality Learning: A New Perspective for Interpretable Machine Learning](https://mlsanity.com/paper/2006.16789/) (2020)
- [Superhuman Artificial Intelligence Can Improve Human Decision Making by Increasing Novelty](https://mlsanity.com/paper/2303.07462/) (2023)
- [Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomena](https://mlsanity.com/paper/2206.05487/) (2022)
- [The role of causality in explainable artificial intelligence](https://mlsanity.com/paper/2309.09901/) (2023)
- [ProtoS-ViT: Visual foundation models for sparse self-explainable classifications](https://mlsanity.com/paper/2406.10025/) (2024)

## Built on by (most cited first, 5 of 5)

- [Explainable Natural Language Processing for Corporate Sustainability Analysis](https://mlsanity.com/paper/2407.17487/) (2024)
- [Revisiting the robustness of post-hoc interpretability methods](https://mlsanity.com/paper/2407.19683/) (2024)
- [XAI4Extremes: An interpretable machine learning framework for understanding extreme-weather precursors under climate change](https://mlsanity.com/paper/2503.08163/) (2025)
- [Explainability matters: The effect of liability rules on the healthcare sector](https://mlsanity.com/paper/2509.17334/) (2025)
- [How are Scientific Concepts Birthed? Typing Rules of Concept Formation in Theoretical Physics Reasoning](https://mlsanity.com/paper/2509.10740/) (2025)

## Abstract (arXiv)

The scientific method is the cornerstone of human progress across all branches of the natural and applied sciences, from understanding the human body to explaining how the universe works. The scientific method is based on identifying systematic rules or principles that describe the phenomenon of interest in a reproducible way that can be validated through experimental evidence. In the era of generative artificial intelligence, there are discussions on how AI systems may discover new knowledge. We argue that human complex reasoning for scientific discovery remains of vital importance, at least before the advent of artificial general intelligence. Yet, AI can be leveraged for scientific discovery via explainable AI. More specifically, knowing the `principles' the AI systems used to make decisions can be a point of contact with domain experts and scientists, that can lead to divergent or convergent views on a given scientific problem. Divergent views may spark further scientific investigations leading to interpretability-guided explanations (IGEs), and possibly to new scientific knowledge. We define this field as Explainable AI for Science, where domain experts -- potentially assisted by generative AI -- formulate scientific hypotheses and explanations based on the interpretability of a predictive AI system.