# ESS-ReduNet: Enhancing Subspace Separability of ReduNet via Dynamic Expansion with Bayesian Inference

arXiv: https://arxiv.org/abs/2411.17961 · HTML: https://mlsanity.com/paper/2411.17961/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training](https://mlsanity.com/paper/2101.12699/) (2021)
- [Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2006.08558/) (2020)
- [Deep Subspace Clustering Networks](https://mlsanity.com/paper/1709.02508/) (2017)
- [ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction](https://mlsanity.com/paper/2105.10446/) (2021)
- [White-Box Transformers via Sparse Rate Reduction](https://mlsanity.com/paper/2306.01129/) (2023)
- [OL'E: Orthogonal Low-rank Embedding, A Plug and Play Geometric Loss for Deep Learning](https://mlsanity.com/paper/1712.01727/) (2017)
- [A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning](https://mlsanity.com/paper/2310.18988/) (2023)
- [White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?](https://mlsanity.com/paper/2311.13110/) (2023)
- [Neural Collaborative Subspace Clustering](https://mlsanity.com/paper/1904.10596/) (2019)
- [Scalable Deep k-Subspace Clustering](https://mlsanity.com/paper/1811.01045/) (2018)
- [Improving the condition number of estimated covariance matrices](https://mlsanity.com/paper/1810.10984/) (2018)

## Built on by (most cited first, 1 of 1)

- [Lightweight Adaptive ReduNet via Hyperspherical Manifold Learning](https://mlsanity.com/paper/2608.20668/) (2026)

## Abstract (arXiv)

ReduNet is a deep neural network model that leverages the principle of maximal coding rate reduction to transform original data samples into a low-dimensional, linear discriminative feature representation. Unlike traditional deep learning frameworks, ReduNet constructs its parameters explicitly layer by layer, with each layer's parameters derived based on the features transformed from the preceding layer. Rather than directly using labels, ReduNet uses the similarity between each category's spanned subspace and the data samples for feature updates at each layer. This may lead to features being updated in the wrong direction, impairing the correct construction of network parameters and reducing the network's convergence speed. To address this issue, based on the geometric interpretation of the network parameters, this paper presents ESS-ReduNet to enhance the separability of each category's subspace by dynamically controlling the expansion of the overall spanned space of the samples. Meanwhile, label knowledge is incorporated with Bayesian inference to encourage the decoupling of subspaces. Finally, stability, as assessed by the condition number, serves as an auxiliary criterion for halting training. Experiments on the ESR, HAR, Covertype, and Gas datasets demonstrate that ESS-ReduNet achieves more than 10x improvement in convergence compared to ReduNet. Notably, on the ESR dataset, the features transformed by ESS-ReduNet achieve a 47% improvement in SVM classification accuracy.