# AICO: Feature Significance Tests for Supervised Learning

arXiv: https://arxiv.org/abs/2506.23396 · HTML: https://mlsanity.com/paper/2506.23396/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 84 references that are in the map.
- First posted on arXiv 29 Jun 2025.

## Builds on (most cited first, 30 of 33)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019)
- [A Survey on the Explainability of Supervised Machine Learning](https://mlsanity.com/paper/2011.07876/) (2020)
- [Problems with Shapley-value-based explanations as feature importance measures](https://mlsanity.com/paper/2002.11097/) (2020)
- [Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection](https://mlsanity.com/paper/1610.02351/) (2016)
- [Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importance](https://mlsanity.com/paper/1905.03151/) (2019)
- [Simple Hardware-Efficient Long Convolutions for Sequence Modeling](https://mlsanity.com/paper/2302.06646/) (2023)
- [A general framework for inference on algorithm-agnostic variable importance](https://mlsanity.com/paper/2004.03683/) (2020)
- [Deep Knockoffs](https://mlsanity.com/paper/1811.06687/) (2018)
- [Conditional Independence Testing using Generative Adversarial Networks](https://mlsanity.com/paper/1907.04068/) (2019)
- [Testing Conditional Independence in Supervised Learning Algorithms](https://mlsanity.com/paper/1901.09917/) (2019)
- [Feature Importance: A Closer Look at Shapley Values and LOCO](https://mlsanity.com/paper/2303.05981/) (2023)
- [Fast and Powerful Conditional Randomization Testing via Distillation](https://mlsanity.com/paper/2006.03980/) (2020)
- [The Missing Indicator Method: From Low to High Dimensions](https://mlsanity.com/paper/2211.09259/) (2022)
- [The Projected Covariance Measure for assumption-lean variable significance testing](https://mlsanity.com/paper/2211.02039/) (2022)
- [Decorrelated Variable Importance](https://mlsanity.com/paper/2111.10853/) (2021)
- [BASED-XAI: Breaking Ablation Studies Down for Explainable Artificial Intelligence](https://mlsanity.com/paper/2207.05566/) (2022)
- [Metropolized Knockoff Sampling](https://mlsanity.com/paper/1903.00434/) (2019)
- [Scalable and Efficient Hypothesis Testing with Random Forests](https://mlsanity.com/paper/1904.07830/) (2019)
- [LOCO Feature Importance Inference without Data Splitting via Minipatch Ensembles](https://mlsanity.com/paper/2206.02088/) (2022)
- [Rank-transformed subsampling: inference for multiple data splitting and exchangeable p-values](https://mlsanity.com/paper/2301.02739/) (2023)
- [Model free variable importance for high dimensional data](https://mlsanity.com/paper/2211.08414/) (2022)
- [Assessing treatment effects in observational data with missing confounders: A comparative study of practical doubly-robust and traditional missing data methods](https://mlsanity.com/paper/2412.15012/) (2024)
- [Multiple Test Functions and Adjusted p-Values for Test Statistics with Discrete Distributions](https://mlsanity.com/paper/1412.0561/) (2014)
- [Improved q-values for discrete uniform and homogeneous tests: a comparative study](https://mlsanity.com/paper/2006.01882/) (2020)

## Abstract (arXiv)

Machine learning is central to modern science, industry, and policy, yet its predictive power often comes at the cost of transparency: we rarely know which input features truly drive a model's predictions. Without such understanding, researchers cannot draw reliable conclusions, practitioners cannot ensure fairness or accountability, and policymakers cannot trust or govern model-based decisions. Existing tools for assessing feature influence are limited; most lack statistical guarantees, and many require costly retraining or surrogate modeling, making them impractical for large modern models. We introduce AICO, a broadly applicable framework that turns model interpretability into an efficient statistical exercise. AICO tests whether each feature genuinely improves predictive performance by masking its information and measuring the resulting change. The method provides exact, finite-sample feature p-values and confidence intervals for feature importance through a simple, non-asymptotic hypothesis testing procedure. It requires no retraining, surrogate modeling, or distributional assumptions, making it feasible for large-scale algorithms. In both controlled experiments and real applications, from credit scoring to mortgage-behavior prediction, AICO reliably identifies the variables that drive model behavior, providing a scalable and statistically principled path toward transparent and trustworthy machine learning.