# Composite Feature Selection using Deep Ensembles

arXiv: https://arxiv.org/abs/2211.00631 · HTML: https://mlsanity.com/paper/2211.00631/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 113 references that are in the map.
- First posted on arXiv 1 Nov 2022.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection](https://mlsanity.com/paper/1610.02351/) (2016)
- [Controlling the false discovery rate via knockoffs](https://mlsanity.com/paper/1404.5609/) (2014)
- [Concrete Autoencoders for Differentiable Feature Selection and Reconstruction](https://mlsanity.com/paper/1901.09346/) (2019)
- [Conditional independence testing based on a nearest-neighbor estimator of conditional mutual information](https://mlsanity.com/paper/1709.01447/) (2017)
- [Model-Powered Conditional Independence Test](https://mlsanity.com/paper/1709.06138/) (2017)
- [Deep Knockoffs](https://mlsanity.com/paper/1811.06687/) (2018)
- [Conditional Independence Testing using Generative Adversarial Networks](https://mlsanity.com/paper/1907.04068/) (2019)
- [Using Attribution to Decode Dataset Bias in Neural Network Models for Chemistry](https://mlsanity.com/paper/1811.11310/) (2018)
- [Group SLOPE - adaptive selection of groups of predictors](https://mlsanity.com/paper/1610.04960/) (2016)
- [Deep Direct Likelihood Knockoffs](https://mlsanity.com/paper/2007.15835/) (2020)
- [The knockoff filter for FDR control in group-sparse and multitask regression](https://mlsanity.com/paper/1602.03589/) (2016)
- [Algorithmic Stability and Generalization of an Unsupervised Feature Selection Algorithm](https://mlsanity.com/paper/2010.09416/) (2020)
- [Deep-gKnock: nonlinear group-feature selection with deep neural network](https://mlsanity.com/paper/1905.10013/) (2019)

## Built on by (most cited first, 11 of 11)

- [Interpretable Deep Clustering for Tabular Data](https://mlsanity.com/paper/2306.04785/) (2023)
- [Multi-Objective Optimization of Performance and Interpretability of Tabular Supervised Machine Learning Models](https://mlsanity.com/paper/2307.08175/) (2023)
- [On the Importance of Step-wise Embeddings for Heterogeneous Clinical Time-Series](https://mlsanity.com/paper/2311.08902/) (2023)
- [Unveiling the Power of Sparse Neural Networks for Feature Selection](https://mlsanity.com/paper/2408.04583/) (2024)
- [Enhancing Representation Learning on High-Dimensional, Small-Size Tabular Data: A Divide and Conquer Method with Ensembled VAEs](https://mlsanity.com/paper/2306.15661/) (2023)
- [Comprehend, Divide, and Conquer: Feature Subspace Exploration via Multi-Agent Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2504.17356/) (2025)
- [Data-Driven Discovery of Feature Groups in Clinical Time Series](https://mlsanity.com/paper/2511.08260/) (2025)
- [EasyFS: an Efficient Model-free Feature Selection Framework via Elastic Transformation of Features](https://mlsanity.com/paper/2402.05954/) (2024)
- [EntryPrune: Neural Network Feature Selection using First Impressions](https://mlsanity.com/paper/2410.02344/) (2024)
- [ICAFS: Inter-Client-Aware Feature Selection for Vertical Federated Learning](https://mlsanity.com/paper/2504.10851/) (2025)
- [Hide&Seek: Learning to Explain in an End-to-End Differentiable Network](https://mlsanity.com/paper/2608.16689/) (2026)

## Abstract (arXiv)

In many real world problems, features do not act alone but in combination with each other. For example, in genomics, diseases might not be caused by any single mutation but require the presence of multiple mutations. Prior work on feature selection either seeks to identify individual features or can only determine relevant groups from a predefined set. We investigate the problem of discovering groups of predictive features without predefined grouping. To do so, we define predictive groups in terms of linear and non-linear interactions between features. We introduce a novel deep learning architecture that uses an ensemble of feature selection models to find predictive groups, without requiring candidate groups to be provided. The selected groups are sparse and exhibit minimum overlap. Furthermore, we propose a new metric to measure similarity between discovered groups and the ground truth. We demonstrate the utility of our model on multiple synthetic tasks and semi-synthetic chemistry datasets, where the ground truth structure is known, as well as an image dataset and a real-world cancer dataset.