# Aggregate Models, Not Explanations: Improving Feature Importance Estimation

arXiv: https://arxiv.org/abs/2602.11760 · HTML: https://mlsanity.com/paper/2602.11760/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 12 Feb 2026.

## Builds on (most cited first, 13 of 13)

- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning](https://mlsanity.com/paper/2012.09816/) (2020)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [A Random Forest Guided Tour](https://mlsanity.com/paper/1511.05741/) (2015)
- [Semiparametric doubly robust targeted double machine learning: a review](https://mlsanity.com/paper/2203.06469/) (2022)
- [Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection](https://mlsanity.com/paper/1610.02351/) (2016)
- [A general framework for inference on algorithm-agnostic variable importance](https://mlsanity.com/paper/2004.03683/) (2020)
- [The Rashomon Importance Distribution: Getting RID of Unstable, Single Model-based Variable Importance](https://mlsanity.com/paper/2309.13775/) (2023)
- [Trees, forests, and impurity-based variable importance](https://mlsanity.com/paper/2001.04295/) (2020)
- [Statistically Valid Variable Importance Assessment through Conditional Permutations](https://mlsanity.com/paper/2309.07593/) (2023)
- [Ultra-marginal Feature Importance: Learning from Data with Causal Guarantees](https://mlsanity.com/paper/2204.09938/) (2022)
- [Measuring Variable Importance in Heterogeneous Treatment Effects with Confidence](https://mlsanity.com/paper/2408.13002/) (2024)
- [A Principled Approach for Defining and Comparing Variable Importance Measures](https://mlsanity.com/paper/2507.17306/) (2025)

## Built on by (most cited first, 1 of 1)

- [CIRCUS: Circuit Consensus under Uncertainty via Stability Ensembles](https://mlsanity.com/paper/2603.00523/) (2026)

## Abstract (arXiv)

Feature-importance methods show promise in transforming machine learning models from predictive engines into tools for scientific discovery. However, due to data sampling and algorithmic stochasticity, expressive models can be unstable, leading to inaccurate variable importance estimates and undermining their utility in critical biomedical applications. Although ensembling offers a solution, deciding whether to explain a single ensemble model or aggregate individual model explanations is difficult due to the nonlinearity of importance measures and remains largely understudied. Our theoretical analysis, developed under assumptions accommodating complex state-of-the-art ML models, reveals that this choice is primarily driven by the model's excess risk. In contrast to prior literature, we show that ensembling at the model level provides more accurate variable-importance estimates, particularly for expressive models, by reducing this leading error term. We validate these findings on classical benchmarks and a large-scale proteomic study from the UK Biobank.