# A Family of Divergence Measures for Evaluating the Reconstruction Quality of Explainable Ensemble Trees

arXiv: https://arxiv.org/abs/2605.19618 · HTML: https://mlsanity.com/paper/2605.19618/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 30 references that are in the map.
- First posted on arXiv 19 May 2026.

## Builds on (most cited first, 5 of 5)

- [On the Use of Random Forest for Two-Sample Testing](https://mlsanity.com/paper/1903.06287/) (2019)
- [Implementing Monte Carlo Tests with P-value Buckets](https://mlsanity.com/paper/1703.09305/) (2017)
- [Double Machine Learning for Partially Linear Mixed-Effects Models with Repeated Measurements](https://mlsanity.com/paper/2108.13657/) (2021)
- [Inference for all variants of the multivariate coefficient of variation in factorial designs](https://mlsanity.com/paper/2301.12009/) (2023)
- [Extending Explainable Ensemble Trees (E2Tree) to regression contexts](https://mlsanity.com/paper/2409.06439/) (2024)

## Abstract (arXiv)

Validating interpretable surrogate models for ensemble learners requires measuring agreement between the ensemble's internal representation and its surrogate approximation, rather than mere association. Correlation-based approaches are scale-invariant and fail to detect systematic discrepancies in co-occurrence structure. We propose a statistical framework grounded in the agreement-association distinction, centered on the normalized Loss of Interpretability (nLoI). Rooted in the Cressie-Read power divergence family with lambda equal to 2, the nLoI admits a closed-form decomposition into within-node and between-node components, providing a unique diagnostic capability to identify precisely where and why reconstruction fails. The framework incorporates four complementary measures capturing distinct structural facets of approximation quality. A unified permutation testing procedure delivers valid inference for all measures within a single resampling pass. Theoretical properties, including boundedness and symmetry, are established for each metric. Monte Carlo simulations and empirical evaluations confirm exact Type I error control and demonstrate that these measures detect reconstruction fidelity gradients invisible to correlation-based alternatives. The framework is developed and illustrated in the context of Explainable Ensemble Trees (E2Tree), and empirical evaluation on three benchmark datasets illustrates the practical utility of the framework.