# Simultaneous Dimensionality Reduction: A Data Efficient Approach for Multimodal Representations Learning

arXiv: https://arxiv.org/abs/2310.04458 · HTML: https://mlsanity.com/paper/2310.04458/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 70 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Builds on (most cited first, 9 of 9)

- [On Deep Multi-View Representation Learning: Objectives and Optimization](https://mlsanity.com/paper/1602.01024/) (2016)
- [Deep Variational Canonical Correlation Analysis](https://mlsanity.com/paper/1610.03454/) (2016)
- [Large-scale neural recordings call for new insights to link brain and behavior](https://mlsanity.com/paper/2103.14662/) (2021)
- [Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models](https://mlsanity.com/paper/2010.13933/) (2020)
- [Multivariate Information Bottleneck](https://mlsanity.com/paper/1301.2270/) (2013)
- [Correlational Neural Networks](https://mlsanity.com/paper/1504.07225/) (2015)
- [Tensor Canonical Correlation Analysis for Multi-view Dimension Reduction](https://mlsanity.com/paper/1502.02330/) (2015)
- [Supervised Dimensionality Reduction for Big Data](https://mlsanity.com/paper/1709.01233/) (2017)
- [Data efficiency, dimensionality reduction, and the generalized symmetric information bottleneck](https://mlsanity.com/paper/2309.05649/) (2023)

## Built on by (most cited first, 6 of 6)

- [Computational Thresholds in Multi-Modal Learning via the Spiked Matrix-Tensor Model](https://mlsanity.com/paper/2506.02664/) (2025)
- [Spectral Thresholds in Correlated Spiked Models and Fundamental Limits of Partial Least Squares](https://mlsanity.com/paper/2510.17561/) (2025)
- [Optimal thresholds and algorithms for a model of multi-modal learning in high dimensions](https://mlsanity.com/paper/2407.03522/) (2024)
- [Accurate Estimation of Mutual Information in High Dimensional Data](https://mlsanity.com/paper/2506.00330/) (2025)
- [Mutual information and task-relevant latent dimensionality](https://mlsanity.com/paper/2602.08105/) (2026)
- [Information bottleneck for learning the phase space of dynamics from high-dimensional experimental data](https://mlsanity.com/paper/2604.24662/) (2026)

## Abstract (arXiv)

We explore two primary classes of approaches to dimensionality reduction (DR): Independent Dimensionality Reduction (IDR) and Simultaneous Dimensionality Reduction (SDR). In IDR methods, of which Principal Components Analysis is a paradigmatic example, each modality is compressed independently, striving to retain as much variation within each modality as possible. In contrast, in SDR, one simultaneously compresses the modalities to maximize the covariation between the reduced descriptions while paying less attention to how much individual variation is preserved. Paradigmatic examples include Partial Least Squares and Canonical Correlations Analysis. Even though these DR methods are a staple of statistics, their relative accuracy and data set size requirements are poorly understood. We introduce a generative linear model to synthesize multimodal data with known variance and covariance structures to examine these questions. We assess the accuracy of the reconstruction of the covariance structures as a function of the number of samples, signal-to-noise ratio, and the number of varying and covarying signals in the data. Using numerical experiments, we demonstrate that linear SDR methods consistently outperform linear IDR methods and yield higher-quality, more succinct reduced-dimensional representations with smaller datasets. Remarkably, regularized CCA can identify low-dimensional weak covarying structures even when the number of samples is much smaller than the dimensionality of the data, which is a regime challenging for all dimensionality reduction methods. Our work corroborates and explains previous observations in the literature that SDR can be more effective in detecting covariation patterns in data. These findings suggest that SDR should be preferred to IDR in real-world data analysis when detecting covariation is more important than preserving variation.