# Variable Importance in High-Dimensional Settings Requires Grouping

arXiv: https://arxiv.org/abs/2312.10858 · HTML: https://mlsanity.com/paper/2312.10858/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 60 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Builds on (most cited first, 16 of 16)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection](https://mlsanity.com/paper/1610.02351/) (2016)
- [Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importance](https://mlsanity.com/paper/1905.03151/) (2019)
- [General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models](https://mlsanity.com/paper/2007.04131/) (2020)
- [A general framework for inference on algorithm-agnostic variable importance](https://mlsanity.com/paper/2004.03683/) (2020)
- [Testing Conditional Independence in Supervised Learning Algorithms](https://mlsanity.com/paper/1901.09917/) (2019)
- [Grouped variable importance with random forests and application to multiple functional data analysis](https://mlsanity.com/paper/1411.4170/) (2014)
- [Grouped Feature Importance and Combined Features Effect Plot](https://mlsanity.com/paper/2104.11688/) (2021)
- [Conditional Feature Importance for Mixed Data](https://mlsanity.com/paper/2210.03047/) (2022)
- [Statistically Valid Variable Importance Assessment through Conditional Permutations](https://mlsanity.com/paper/2309.07593/) (2023)
- [Lazy Estimation of Variable Importance for Large Neural Networks](https://mlsanity.com/paper/2207.09097/) (2022)
- [Multimodal Feature Fusion for Video Advertisements Tagging Via Stacking Ensemble](https://mlsanity.com/paper/2108.00679/) (2021)
- [Understanding Learned Models by Identifying Important Features at the Right Resolution](https://mlsanity.com/paper/1811.07279/) (2018)

## Built on by (most cited first, 4 of 4)

- [Metric Learning Encoding Models: A Multivariate Framework for Interpreting Neural Representations](https://mlsanity.com/paper/2402.11608/) (2024)
- [Measuring Variable Importance in Heterogeneous Treatment Effects with Confidence](https://mlsanity.com/paper/2408.13002/) (2024)
- [Top-k Feature Importance Ranking](https://mlsanity.com/paper/2509.15420/) (2025)
- [Hierarchical Variable Importance with Statistical Control for Medical Data-Based Prediction](https://mlsanity.com/paper/2508.08724/) (2025)

## Abstract (arXiv)

Explaining the decision process of machine learning algorithms is nowadays crucial for both model's performance enhancement and human comprehension. This can be achieved by assessing the variable importance of single variables, even for high-capacity non-linear methods, e.g. Deep Neural Networks (DNNs). While only removal-based approaches, such as Permutation Importance (PI), can bring statistical validity, they return misleading results when variables are correlated. Conditional Permutation Importance (CPI) bypasses PI's limitations in such cases. However, in high-dimensional settings, where high correlations between the variables cancel their conditional importance, the use of CPI as well as other methods leads to unreliable results, besides prohibitive computation costs. Grouping variables statistically via clustering or some prior knowledge gains some power back and leads to better interpretations. In this work, we introduce BCPI (Block-Based Conditional Permutation Importance), a new generic framework for variable importance computation with statistical guarantees handling both single and group cases. Furthermore, as handling groups with high cardinality (such as a set of observations of a given modality) are both time-consuming and resource-intensive, we also introduce a new stacking approach extending the DNN architecture with sub-linear layers adapted to the group structure. We show that the ensuing approach extended with stacking controls the type-I error even with highly-correlated groups and shows top accuracy across benchmarks. Furthermore, we perform a real-world data analysis in a large-scale medical dataset where we aim to show the consistency between our results and the literature for a biomarker prediction.