# Validity of Feature Importance in Low-Performing Machine Learning for Tabular Biomedical Data

arXiv: https://arxiv.org/abs/2409.13342 · HTML: https://mlsanity.com/paper/2409.13342/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 57 references that are in the map.
- First posted on arXiv 20 Sep 2024.

## Builds on (most cited first, 11 of 11)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importance](https://mlsanity.com/paper/1905.03151/) (2019)
- [Feature-Wise Bias Amplification](https://mlsanity.com/paper/1812.08999/) (2018)
- [Addressing Fairness, Bias and Class Imbalance in Machine Learning: the FBI-loss](https://mlsanity.com/paper/2105.06345/) (2021)
- [Rethinking Class Imbalance in Machine Learning](https://mlsanity.com/paper/2305.03900/) (2023)

## Abstract (arXiv)

In tabular biomedical data analysis, tuning models to high accuracy is considered a prerequisite for discussing feature importance, as medical practitioners expect the validity of feature importance to correlate with performance. In this work, we challenge the prevailing belief, showing that low-performing models may also be used for feature importance. We propose experiments to observe changes in feature rank as performance degrades sequentially. Using three synthetic datasets and six real biomedical datasets, we compare the rank of features from full datasets to those with reduced sample sizes (data cutting) or fewer features (feature cutting). In synthetic datasets, feature cutting does not change feature rank, while data cutting shows higher discrepancies with lower performance. In real datasets, feature cutting shows similar or smaller changes than data cutting, though some datasets exhibit the opposite. When feature interactions are controlled by removing correlations, feature cutting consistently shows better stability. By analyzing the distribution of feature importance values and theoretically examining the probability that the model cannot distinguish feature importance between features, we reveal that models can still distinguish feature importance despite performance degradation through feature cutting, but not through data cutting. We conclude that the validity of feature importance can be maintained even at low performance levels if the data size is adequate, which is a significant factor contributing to suboptimal performance in tabular medical data analysis. This paper demonstrates the potential for utilizing feature importance analysis alongside statistical analysis to compare features relatively, even when classifier performance is not satisfactory.