# The Trade-off Between Covariate Dependence and Latent Structure in Representation Learning

arXiv: https://arxiv.org/abs/2608.16245 · HTML: https://mlsanity.com/paper/2608.16245/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 57 references that are in the map.
- First posted on arXiv 17 Aug 2026.

## Builds on (most cited first, 20 of 20)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning](https://mlsanity.com/paper/1805.08651/) (2018)
- [Weakly-Supervised Disentanglement Without Compromises](https://mlsanity.com/paper/2002.02886/) (2020)
- [Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations](https://mlsanity.com/paper/1705.08841/) (2017)
- [Optimal whitening and decorrelation](https://mlsanity.com/paper/1512.00809/) (2015)
- [On Disentangled Representations Learned From Correlated Data](https://mlsanity.com/paper/2006.07886/) (2020)
- [Learning Latent Subspaces in Variational Autoencoders](https://mlsanity.com/paper/1812.06190/) (2018)
- [Disentanglement of Correlated Factors via Hausdorff Factorized Support](https://mlsanity.com/paper/2210.07347/) (2022)
- [Learning Independent Features with Adversarial Nets for Non-linear ICA](https://mlsanity.com/paper/1710.05050/) (2017)
- [Group Factor Analysis](https://mlsanity.com/paper/1411.5799/) (2014)
- [sJIVE: Supervised Joint and Individual Variation Explained](https://mlsanity.com/paper/2102.13278/) (2021)
- [A Two-Step Disentanglement Method](https://mlsanity.com/paper/1709.00199/) (2017)
- [Disentanglement and Generalization Under Correlation Shifts](https://mlsanity.com/paper/2112.14754/) (2021)
- [Bayesian Joint Additive Factor Models for Multiview Learning](https://mlsanity.com/paper/2406.00778/) (2024)
- [Encoding Domain Knowledge in Multi-view Latent Variable Models: A Bayesian Approach with Structured Sparsity](https://mlsanity.com/paper/2204.06242/) (2022)
- [Structural Equation-VAE: Disentangled Latent Representations for Tabular Data](https://mlsanity.com/paper/2508.06347/) (2025)

## Abstract (arXiv)

Disentangled representation learning seeks latent representations whose indicidual dimensions each align with a distinct covariate. Unsupervised approaches typically target latent dimension independence, yet this gives no guarantee that the resulting dimensions align with semantically meaningful covariates. Supervised approaches structure the latent space using observed covariates, but under correlated covariates they cannot simultaneously control one-to-one latent-covariate alignment and latent independence. We introduce a unified, supervised framework that couples latent dimension-covariate dependence with constraints on the latent structure. Within this framework, we show an inherent trade-off, where enforcing latent independence or exclusive one-to-one latent-covariate dependence comes at a provable cost in latent-covariate alignment. We prove that the resulting disentanglement regimes are ordered by the strength of that alignment. Each regime admits a closed-form transformation of the latent space. We apply these transformations post-hoc to realign the representations of pretrained models such as CLIP, DINOv2, and ViT, and we fold them into the inference of informed factor analysis (iFA), a probabilistic model with covariate-informed factors. On simulated and real multi-omics data, we show that both post-hoc alignment and iFA enable controllability of structured latent representations.