# On Interpretable Approaches to Cluster, Classify and Represent Multi-Subspace Data via Minimum Lossy Coding Length based on Rate-Distortion Theory

arXiv: https://arxiv.org/abs/2302.10383 · HTML: https://mlsanity.com/paper/2302.10383/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 68 references that are in the map.
- First posted on arXiv 21 Feb 2023.

## Builds on (most cited first, 19 of 19)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [A General Theory of Equivariant CNNs on Homogeneous Spaces](https://mlsanity.com/paper/1811.02017/) (2018)
- [Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2006.08558/) (2020)
- [ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction](https://mlsanity.com/paper/2105.10446/) (2021)
- [Deep Isometric Learning for Visual Recognition](https://mlsanity.com/paper/2006.16992/) (2020)
- [OL'E: Orthogonal Low-rank Embedding, A Plug and Play Geometric Loss for Deep Learning](https://mlsanity.com/paper/1712.01727/) (2017)
- [A Rate-Distortion Framework for Explaining Neural Network Decisions](https://mlsanity.com/paper/1905.11092/) (2019)
- [Learning a Self-Expressive Network for Subspace Clustering](https://mlsanity.com/paper/2110.04318/) (2021)
- [How Generative Adversarial Networks and Their Variants Work: An Overview](https://mlsanity.com/paper/1711.05914/) (2017)
- [Incremental Learning of Structured Memory via Closed-Loop Transcription](https://mlsanity.com/paper/2202.05411/) (2022)
- [Multimodal Image Synthesis with Conditional Implicit Maximum Likelihood Estimation](https://mlsanity.com/paper/2004.03590/) (2020)
- [LogDet Rank Minimization with Application to Subspace Clustering](https://mlsanity.com/paper/1507.00908/) (2015)
- [Efficient Maximal Coding Rate Reduction by Variational Forms](https://mlsanity.com/paper/2204.00077/) (2022)

## Abstract (arXiv)

To cluster, classify and represent are three fundamental objectives of learning from high-dimensional data with intrinsic structure. To this end, this paper introduces three interpretable approaches, i.e., segmentation (clustering) via the Minimum Lossy Coding Length criterion, classification via the Minimum Incremental Coding Length criterion and representation via the Maximal Coding Rate Reduction criterion. These are derived based on the lossy data coding and compression framework from the principle of rate distortion in information theory. These algorithms are particularly suitable for dealing with finite-sample data (allowed to be sparse or almost degenerate) of mixed Gaussian distributions or subspaces. The theoretical value and attractive features of these methods are summarized by comparison with other learning methods or evaluation criteria. This summary note aims to provide a theoretical guide to researchers (also engineers) interested in understanding 'white-box' machine (deep) learning methods.