# Automatic feature selection and weighting in molecular systems using Differentiable Information Imbalance

arXiv: https://arxiv.org/abs/2411.00851 · HTML: https://mlsanity.com/paper/2411.00851/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 84 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Builds on (most cited first, 14 of 14)

- [Estimating the intrinsic dimension of datasets by a minimal neighborhood information](https://mlsanity.com/paper/1803.06992/) (2018)
- [DScribe: Library of Descriptors for Machine Learning in Materials Science](https://mlsanity.com/paper/1904.08875/) (2019)
- [Relief-Based Feature Selection: Introduction and Review](https://mlsanity.com/paper/1711.08421/) (2017)
- [Survey on Feature Selection](https://mlsanity.com/paper/1510.02892/) (2015)
- [Ranking the information content of distance measures](https://mlsanity.com/paper/2104.15079/) (2021)
- [DADApy: Distance-based Analysis of DAta-manifolds in Python](https://mlsanity.com/paper/2205.03373/) (2022)
- [Benchmarking Relief-Based Feature Selection Methods for Bioinformatics Data Mining](https://mlsanity.com/paper/1711.08477/) (2017)
- [Automatic topography of high-dimensional data sets by non-parametric Density Peak clustering](https://mlsanity.com/paper/1802.10549/) (2018)
- [Optimal approximation for unconstrained non-submodular minimization](https://mlsanity.com/paper/1905.12145/) (2019)
- [Tensor-reduced atomic density representations](https://mlsanity.com/paper/2210.01705/) (2022)
- [Supervised feature selection with orthogonal regression and feature weighting](https://mlsanity.com/paper/1910.03787/) (2019)
- [Filter Methods for Feature Selection in Supervised Machine Learning Applications -- Review and Benchmark](https://mlsanity.com/paper/2111.12140/) (2021)
- [Analyzing the impact of feature selection on the accuracy of heart disease prediction](https://mlsanity.com/paper/2206.03239/) (2022)
- [Maximally informative feature selection using Information Imbalance: Application to COVID-19 severity prediction](https://mlsanity.com/paper/2404.14275/) (2024)

## Built on by (most cited first, 5 of 5)

- [A quantitative analysis of semantic information in deep representations of text and images](https://mlsanity.com/paper/2505.17101/) (2025)
- [Differential syntactic and semantic encoding in LLMs](https://mlsanity.com/paper/2601.04765/) (2026)
- [The Effect of Label Noise on the Information Content of Neural Representations](https://mlsanity.com/paper/2510.06401/) (2025)
- [ARISE: An adaptive residual-informed stability ensemble for feature selection in small-sample biomedical omics](https://mlsanity.com/paper/2608.14866/) (2026)
- [A temporal graph dissimilarity measure capturing epidemic spreading behavior](https://mlsanity.com/paper/2610.16456/) (2026)

## Abstract (arXiv)

Feature selection is essential in the analysis of molecular systems and many other fields, but several uncertainties remain: What is the optimal number of features for a simplified, interpretable model that retains essential information? How should features with different units be aligned, and how should their relative importance be weighted? Here, we introduce the Differentiable Information Imbalance (DII), an automated method to rank information content between sets of features. Using distances in a ground truth feature space, DII identifies a low-dimensional subset of features that best preserves these relationships. Each feature is scaled by a weight, which is optimized by minimizing the DII through gradient descent. This allows simultaneously performing unit alignment and relative importance scaling, while preserving interpretability. DII can also produce sparse solutions and determine the optimal size of the reduced feature space. We demonstrate the usefulness of this approach on two benchmark molecular problems: (1) identifying collective variables that describe conformations of a biomolecule, and (2) selecting features for training a machine-learning force field. These results show the potential of DII in addressing feature selection challenges and optimizing dimensionality in various applications. The method is available in the Python library DADApy.