# Hypothesis Testing and Machine Learning: Interpreting Variable Effects in Deep Artificial Neural Networks using Cohen's f2

arXiv: https://arxiv.org/abs/2302.01407 · HTML: https://mlsanity.com/paper/2302.01407/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 104 references that are in the map.
- First posted on arXiv 2 Feb 2023.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models](https://mlsanity.com/paper/1612.08468/) (2016)
- [Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences](https://mlsanity.com/paper/1712.00547/) (2017)
- [Interpretability of machine learning based prediction models in healthcare](https://mlsanity.com/paper/2002.08596/) (2020)
- [What Does Explainable AI Really Mean? A New Conceptualization of Perspectives](https://mlsanity.com/paper/1710.00794/) (2017)
- [dalex: Responsible Machine Learning with Interactive Explainability and Fairness in Python](https://mlsanity.com/paper/2012.14406/) (2020)
- [Stop ordering machine learning algorithms by their explainability! A user-centered investigation of performance and explainability](https://mlsanity.com/paper/2206.10610/) (2022)
- [Shapley variable importance clouds for interpretable machine learning](https://mlsanity.com/paper/2110.02484/) (2021)
- [Interpreting Deep Neural Networks Through Variable Importance](https://mlsanity.com/paper/1901.09839/) (2019)

## Abstract (arXiv)

Deep artificial neural networks show high predictive performance in many fields, but they do not afford statistical inferences and their black-box operations are too complicated for humans to comprehend. Because positing that a relationship exists is often more important than prediction in scientific experiments and research models, machine learning is far less frequently used than inferential statistics. Additionally, statistics calls for improving the test of theory by showing the magnitude of the phenomena being studied. This article extends current XAI methods and develops a model agnostic hypothesis testing framework for machine learning. First, Fisher's variable permutation algorithm is tweaked to compute an effect size measure equivalent to Cohen's f2 for OLS regression models. Second, the Mann-Kendall test of monotonicity and the Theil-Sen estimator is applied to Apley's accumulated local effect plots to specify a variable's direction of influence and statistical significance. The usefulness of this approach is demonstrated on an artificial data set and a social survey with a Python sandbox implementation.