# Correlation and variable importance in random forests

arXiv: https://arxiv.org/abs/1310.5726 · HTML: https://mlsanity.com/paper/1310.5726/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2013.

## Built on by (most cited first, 30 of 41)

- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [Machine learning pipeline for battery state of health estimation](https://mlsanity.com/paper/2102.00837/) (2021)
- [Comparing Explanation Methods for Traditional Machine Learning Models Part 1: An Overview of Current Methods and Quantifying Their Disagreement](https://mlsanity.com/paper/2211.08943/) (2022)
- [MDA for random forests: inconsistency, and a practical solution via the Sobol-MDA](https://mlsanity.com/paper/2102.13347/) (2021)
- [LEAF: Navigating Concept Drift in Cellular Networks](https://mlsanity.com/paper/2109.03011/) (2021)
- [BA-Net: Bridge Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/2112.04150/) (2021)
- [Incremental Permutation Feature Importance (iPFI): Towards Online Explanations on Data Streams](https://mlsanity.com/paper/2209.01939/) (2022)
- [Feature Inference Attack on Shapley Values](https://mlsanity.com/paper/2407.11359/) (2024)
- [Feature Importance Explanations for Temporal Black-Box Models](https://mlsanity.com/paper/2102.11934/) (2021)
- [Feature Importance in Gradient Boosting Trees with Cross-Validation Feature Selection](https://mlsanity.com/paper/2109.05468/) (2021)
- [A Generic Shared Attention Mechanism for Various Backbone Neural Networks](https://mlsanity.com/paper/2210.16101/) (2022)
- [Exploring the cloud of feature interaction scores in a Rashomon set](https://mlsanity.com/paper/2305.10181/) (2023)
- [Quantile-constrained Wasserstein projections for robust interpretability of numerical and machine learning models](https://mlsanity.com/paper/2209.11539/) (2022)
- [FedScore: A privacy-preserving framework for federated scoring system development](https://mlsanity.com/paper/2303.00282/) (2023)
- [Time Series Feature Redundancy Paradox: An Empirical Study Based on Mortgage Default Prediction](https://mlsanity.com/paper/2501.00034/) (2024)
- [A model-free feature selection technique of feature screening and random forest based recursive feature elimination](https://mlsanity.com/paper/2302.07449/) (2023)
- [Feature Selection for Network Intrusion Detection](https://mlsanity.com/paper/2411.11603/) (2024)
- [Wind power predictions from nowcasts to 4-hour forecasts: a learning approach with variable selection](https://mlsanity.com/paper/2204.09362/) (2022)
- [Comparing Explanation Methods for Traditional Machine Learning Models Part 2: Quantifying Model Explainability Faithfulness and Improvements with Dimensionality Reduction](https://mlsanity.com/paper/2211.10378/) (2022)
- [Can predictive models be used for causal inference?](https://mlsanity.com/paper/2306.10551/) (2023)
- [MMD-based Variable Importance for Distributional Random Forest](https://mlsanity.com/paper/2310.12115/) (2023)
- [A Subspace-based Approach for Dimensionality Reduction and Important Variable Selection](https://mlsanity.com/paper/2106.01584/) (2021)
- [FACT: High-Dimensional Random Forests Inference](https://mlsanity.com/paper/2207.01678/) (2022)
- [A Pipeline for Data-Driven Learning of Topological Features with Applications to Protein Stability Prediction](https://mlsanity.com/paper/2408.04847/) (2024)
- [Stability Selection via Variable Decorrelation](https://mlsanity.com/paper/2505.20864/) (2025)
- [Decorrelated feature importance from local sample weighting](https://mlsanity.com/paper/2508.06337/) (2025)
- [Layer-based Composite Reputation Bootstrapping](https://mlsanity.com/paper/2102.09951/) (2021)
- [SalienTrack: providing salient information for semi-automated self-tracking feedback with model explanations](https://mlsanity.com/paper/2109.10231/) (2021)
- [Interpretable Models via Pairwise permutations algorithm](https://mlsanity.com/paper/2111.09145/) (2021)
- [A machine learning approach to predict the structural and magnetic properties of Heusler alloy families](https://mlsanity.com/paper/2208.12705/) (2022)

## Abstract (arXiv)

This paper is about variable selection with the random forests algorithm in presence of correlated predictors. In high-dimensional regression or classification frameworks, variable selection is a difficult task, that becomes even more challenging in the presence of highly correlated predictors. Firstly we provide a theoretical study of the permutation importance measure for an additive regression model. This allows us to describe how the correlation between predictors impacts the permutation importance. Our results motivate the use of the Recursive Feature Elimination (RFE) algorithm for variable selection in this context. This algorithm recursively eliminates the variables using permutation importance measure as a ranking criterion. Next various simulation experiments illustrate the efficiency of the RFE algorithm for selecting a small number of variables together with a good prediction error. Finally, this selection algorithm is tested on the Landsat Satellite data from the UCI Machine Learning Repository.