# Optimizing model-agnostic Random Subspace ensembles

arXiv: https://arxiv.org/abs/2109.03099 · HTML: https://mlsanity.com/paper/2109.03099/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 44 references that are in the map.
- First posted on arXiv 7 Sep 2021.

## Builds on (most cited first, 8 of 8)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparison](https://mlsanity.com/paper/1703.00512/) (2017)
- [Concrete Autoencoders for Differentiable Feature Selection and Reconstruction](https://mlsanity.com/paper/1901.09346/) (2019)
- [Feature Selection: A Data Perspective](https://mlsanity.com/paper/1601.07996/) (2016)
- [Locally Sparse Neural Networks for Tabular Biomedical Data](https://mlsanity.com/paper/2106.06468/) (2021)
- [RaSE: Random Subspace Ensemble Classification](https://mlsanity.com/paper/2006.08855/) (2020)
- [Differentiable Feature Selection, a Reparameterization Approach](https://mlsanity.com/paper/2107.10030/) (2021)

## Built on by (most cited first, 3 of 3)

- [EFI: A Toolbox for Feature Importance Fusion and Interpretation in Python](https://mlsanity.com/paper/2208.04343/) (2022)
- [POSSE-kNN: Pathwise Out-of-Bag Selected Subspace Ensembles for Binary Classification](https://mlsanity.com/paper/2211.11278/) (2022)
- [Conjunction Subspaces Test for Conformal and Selective Classification](https://mlsanity.com/paper/2410.12297/) (2024)

## Abstract (arXiv)

This paper presents a model-agnostic ensemble approach for supervised learning. The proposed approach is based on a parametric version of Random Subspace, in which each base model is learned from a feature subset sampled according to a Bernoulli distribution. Parameter optimization is performed using gradient descent and is rendered tractable by using an importance sampling approach that circumvents frequent re-training of the base models after each gradient descent step. The degree of randomization in our parametric Random Subspace is thus automatically tuned through the optimization of the feature selection probabilities. This is an advantage over the standard Random Subspace approach, where the degree of randomization is controlled by a hyper-parameter. Furthermore, the optimized feature selection probabilities can be interpreted as feature importance scores. Our algorithm can also easily incorporate any differentiable regularization term to impose constraints on these importance scores.