# Stochastic Mutual Information Gradient Estimation for Dimensionality Reduction Networks

arXiv: https://arxiv.org/abs/2105.00191 · HTML: https://mlsanity.com/paper/2105.00191/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 51 references that are in the map.
- First posted on arXiv 1 May 2021.

## Builds on (most cited first, 8 of 8)

- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017)
- [Learning Disentangled Representations via Mutual Information Estimation](https://mlsanity.com/paper/1912.03915/) (2019)
- [Estimating Mutual Information for Discrete-Continuous Mixtures](https://mlsanity.com/paper/1709.06212/) (2017)
- [Deep Learning for Channel Coding via Neural Mutual Information Estimation](https://mlsanity.com/paper/1903.02865/) (2019)
- [Mutual Information Gradient Estimation for Representation Learning](https://mlsanity.com/paper/2005.01123/) (2020)
- [Information Theoretic Feature Transformation Learning for Brain Interfaces](https://mlsanity.com/paper/1903.12235/) (2019)

## Built on by (most cited first, 2 of 2)

- [Entropy-based Guidance of Deep Neural Networks for Accelerated Convergence and Improved Performance](https://mlsanity.com/paper/2308.14938/) (2023)
- [TS-MoCo: Time-Series Momentum Contrast for Self-Supervised Physiological Representation Learning](https://mlsanity.com/paper/2306.06522/) (2023)

## Abstract (arXiv)

Feature ranking and selection is a widely used approach in various applications of supervised dimensionality reduction in discriminative machine learning. Nevertheless there exists significant evidence on feature ranking and selection algorithms based on any criterion leading to potentially sub-optimal solutions for class separability. In that regard, we introduce emerging information theoretic feature transformation protocols as an end-to-end neural network training approach. We present a dimensionality reduction network (MMINet) training procedure based on the stochastic estimate of the mutual information gradient. The network projects high-dimensional features onto an output feature space where lower dimensional representations of features carry maximum mutual information with their associated class labels. Furthermore, we formulate the training objective to be estimated non-parametrically with no distributional assumptions. We experimentally evaluate our method with applications to high-dimensional biological data sets, and relate it to conventional feature selection algorithms to form a special case of our approach.