# Significance tests of feature relevance for a black-box learner

arXiv: https://arxiv.org/abs/2103.04985 · HTML: https://mlsanity.com/paper/2103.04985/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 64 references that are in the map.
- First posted on arXiv 2 Mar 2021.

## Builds on (most cited first, 13 of 13)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition](https://mlsanity.com/paper/1503.02101/) (2015)
- [Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis](https://mlsanity.com/paper/1702.03849/) (2017)
- [Universal Inference](https://mlsanity.com/paper/1912.11436/) (2019)
- [Facial Expression Recognition using Convolutional Neural Networks: State of the Art](https://mlsanity.com/paper/1612.02903/) (2016)
- [The Holdout Randomization Test for Feature Selection in Black Box Models](https://mlsanity.com/paper/1811.00645/) (2018)

## Built on by (most cited first, 15 of 15)

- [Feature Importance: A Closer Look at Shapley Values and LOCO](https://mlsanity.com/paper/2303.05981/) (2023)
- [Decorrelated Variable Importance](https://mlsanity.com/paper/2111.10853/) (2021)
- [Conditional independence testing under misspecified inductive biases](https://mlsanity.com/paper/2307.02520/) (2023)
- [Boosting Data Analytics With Synthetic Volume Expansion](https://mlsanity.com/paper/2310.17848/) (2023)
- [General Frameworks for Conditional Two-Sample Testing](https://mlsanity.com/paper/2410.16636/) (2024)
- [CeCNN: Copula-enhanced convolutional neural networks in joint prediction of refraction error and axial length based on ultra-widefield fundus images](https://mlsanity.com/paper/2311.03967/) (2023)
- [Deep Nonparametric Conditional Independence Tests for Images](https://mlsanity.com/paper/2411.06140/) (2024)
- [iLOCO: Distribution-Free Inference for Feature Interactions](https://mlsanity.com/paper/2502.06661/) (2025)
- [Perturbation-Assisted Sample Synthesis: A Novel Approach for Uncertainty Quantification](https://mlsanity.com/paper/2305.18671/) (2023)
- [Embedded Conditional Independence Tests for Large Language Model Generated Text with an Application to German Parliament Speeches](https://mlsanity.com/paper/2609.00946/) (2026)
- [Data-Adaptive Discriminative Feature Localization with Statistically Guaranteed Interpretation](https://mlsanity.com/paper/2211.10061/) (2022)
- [A Two-Sample Test of Text Generation Similarity](https://mlsanity.com/paper/2505.05269/) (2025)
- [Fast Flow Matching based Conditional Independence Tests for Causal Discovery](https://mlsanity.com/paper/2602.08315/) (2026)
- [Generative Score Inference for Multimodal Data](https://mlsanity.com/paper/2603.26349/) (2026)
- [Cross-Fitting Under Nonregularity: Normality and Inference via Locality](https://mlsanity.com/paper/2610.02944/) (2026)

## Abstract (arXiv)

An exciting recent development is the uptake of deep neural networks in many scientific fields, where the main objective is outcome prediction with the black-box nature. Significance testing is promising to address the black-box issue and explore novel scientific insights and interpretation of the decision-making process based on a deep learning model. However, testing for a neural network poses a challenge because of its black-box nature and unknown limiting distributions of parameter estimates while existing methods require strong assumptions or excessive computation. In this article, we derive one-split and two-split tests relaxing the assumptions and computational complexity of existing black-box tests and extending to examine the significance of a collection of features of interest in a dataset of possibly a complex type such as an image. The one-split test estimates and evaluates a black-box model based on estimation and inference subsets through sample splitting and data perturbation. The two-split test further splits the inference subset into two but require no perturbation. Also, we develop their combined versions by aggregating the p-values based on repeated sample splitting. By deflating the bias-sd-ratio, we establish asymptotic null distributions of the test statistics and the consistency in terms of Type II error. Numerically, we demonstrate the utility of the proposed tests on seven simulated examples and six real datasets. Accompanying this paper is our Python library dnn-inference (https://dnn-inference.readthedocs.io/en/latest/) that implements the proposed tests.