# MDA for random forests: inconsistency, and a practical solution via the Sobol-MDA

arXiv: https://arxiv.org/abs/2102.13347 · HTML: https://mlsanity.com/paper/2102.13347/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 66 references that are in the map.
- First posted on arXiv 26 Feb 2021.

## Builds on (most cited first, 18 of 18)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Consistent Individualized Feature Attribution for Tree Ensembles](https://mlsanity.com/paper/1802.03888/) (2018)
- [Explaining individual predictions when features are dependent: More accurate approximations to Shapley values](https://mlsanity.com/paper/1903.10464/) (2019)
- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importance](https://mlsanity.com/paper/1905.03151/) (2019)
- [Consistency of random forests](https://mlsanity.com/paper/1405.2881/) (2014)
- [Understanding Random Forests: From Theory to Practice](https://mlsanity.com/paper/1407.7502/) (2014)
- [A general framework for inference on algorithm-agnostic variable importance](https://mlsanity.com/paper/2004.03683/) (2020)
- [Correlation and variable importance in random forests](https://mlsanity.com/paper/1310.5726/) (2013)
- [Unbiased Measurement of Feature Importance in Tree-Based Methods](https://mlsanity.com/paper/1903.05179/) (2019)
- [A Debiased MDI Feature Importance Measure for Random Forests](https://mlsanity.com/paper/1906.10845/) (2019)
- [Asymptotic Distributions and Rates of Convergence for Random Forests via Generalized U-statistics](https://mlsanity.com/paper/1905.10651/) (2019)
- [Grouped variable importance with random forests and application to multiple functional data analysis](https://mlsanity.com/paper/1411.4170/) (2014)
- [Trees, forests, and impurity-based variable importance](https://mlsanity.com/paper/2001.04295/) (2020)
- [Unbiased variable importance for random forests](https://mlsanity.com/paper/2003.02106/) (2020)
- [Getting Better from Worse: Augmented Bagging and a Cautionary Tale of Variable Importance](https://mlsanity.com/paper/2003.03629/) (2020)
- [Understanding Global Feature Contributions With Additive Importance Measures](https://mlsanity.com/paper/2004.00668/) (2020)

## Built on by (most cited first, 10 of 10)

- [Feature Importance: A Closer Look at Shapley Values and LOCO](https://mlsanity.com/paper/2303.05981/) (2023)
- [SHAFF: Fast and consistent SHApley eFfect estimates via random Forests](https://mlsanity.com/paper/2105.11724/) (2021)
- [Fast Interpretable Greedy-Tree Sums](https://mlsanity.com/paper/2201.11931/) (2022)
- [Variable importance for causal forests: breaking down the heterogeneity of treatment effects](https://mlsanity.com/paper/2308.03369/) (2023)
- [Consistent Sufficient Explanations and Minimal Local Rules for explaining regression and classification models](https://mlsanity.com/paper/2111.04658/) (2021)
- [Quantile-constrained Wasserstein projections for robust interpretability of numerical and machine learning models](https://mlsanity.com/paper/2209.11539/) (2022)
- [Shapley Curves: A Smoothing Perspective](https://mlsanity.com/paper/2211.13289/) (2022)
- [FACT: High-Dimensional Random Forests Inference](https://mlsanity.com/paper/2207.01678/) (2022)
- [Local and Regional Counterfactual Rules: Summarized and Robust Recourses](https://mlsanity.com/paper/2209.14568/) (2022)
- [Transfer learning for causal forest](https://mlsanity.com/paper/2606.07693/) (2026)

## Abstract (arXiv)

Variable importance measures are the main tools to analyze the black-box mechanisms of random forests. Although the mean decrease accuracy (MDA) is widely accepted as the most efficient variable importance measure for random forests, little is known about its statistical properties. In fact, the definition of MDA varies across the main random forest software. In this article, our objective is to rigorously analyze the behavior of the main MDA implementations. Consequently, we mathematically formalize the various implemented MDA algorithms, and then establish their limits when the sample size increases. This asymptotic analysis reveals that these MDA versions differ as importance measures, since they converge towards different quantities. More importantly, we break down these limits into three components: the first two terms are related to Sobol indices, which are well-defined measures of a covariate contribution to the response variance, widely used in the sensitivity analysis field, as opposed to the third term, whose value increases with dependence within covariates. Thus, we theoretically demonstrate that the MDA does not target the right quantity to detect influential covariates in a dependent setting, a fact that has already been noticed experimentally. To address this issue, we define a new importance measure for random forests, the Sobol-MDA, which fixes the flaws of the original MDA, and consistently estimates the accuracy decrease of the forest retrained without a given covariate, but with an efficient computational cost. The Sobol-MDA empirically outperforms its competitors on both simulated and real data for variable selection. An open source implementation in R and C++ is available online.