# Dependence Induced Representations

arXiv: https://arxiv.org/abs/2411.15328 · HTML: https://mlsanity.com/paper/2411.15328/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 19 references that are in the map.
- First posted on arXiv 22 Nov 2024.

## Builds on (most cited first, 7 of 7)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Matryoshka Representation Learning](https://mlsanity.com/paper/2205.13147/) (2022)
- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [Neural collapse with unconstrained features](https://mlsanity.com/paper/2011.11619/) (2020)
- [On Universal Features for High-Dimensional Learning and Inference](https://mlsanity.com/paper/1911.09105/) (2019)
- [Neural Feature Learning in Function Space](https://mlsanity.com/paper/2309.10140/) (2023)

## Built on by (most cited first, 2 of 2)

- [A Statistical Theory of Contrastive Learning via Approximate Sufficient Statistics](https://mlsanity.com/paper/2503.17538/) (2025)
- [Separable Computation of Information Measures](https://mlsanity.com/paper/2501.15301/) (2025)

## Abstract (arXiv)

We study the problem of learning feature representations from a pair of random variables, where we focus on the representations that are induced by their dependence. We provide sufficient and necessary conditions for such dependence induced representations, and illustrate their connections to Hirschfeld--Gebelein--R'enyi (HGR) maximal correlation functions and minimal sufficient statistics. We characterize a large family of loss functions that can learn dependence induced representations, including cross entropy, hinge loss, and their regularized variants. In particular, we show that the features learned from this family can be expressed as the composition of a loss-dependent function and the maximal correlation function, which reveals a key connection between representations learned from different losses. Our development also gives a statistical interpretation of the neural collapse phenomenon observed in deep classifiers. Finally, we present the learning design based on the feature separation, which allows hyperparameter tuning during inference.