# An Online Riemannian PCA for Stochastic Canonical Correlation Analysis

arXiv: https://arxiv.org/abs/2106.07479 · HTML: https://mlsanity.com/paper/2106.07479/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 49 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [Empirical Risk Minimization under Fairness Constraints](https://mlsanity.com/paper/1802.08626/) (2018)
- [Riemannian Adaptive Optimization Methods](https://mlsanity.com/paper/1810.00760/) (2018)
- [On Deep Multi-View Representation Learning: Objectives and Optimization](https://mlsanity.com/paper/1602.01024/) (2016)
- [Hypercolumns for Object Segmentation and Fine-grained Localization](https://mlsanity.com/paper/1411.5752/) (2014)
- [Manopt, a Matlab toolbox for optimization on manifolds](https://mlsanity.com/paper/1308.5200/) (2013)
- [Beyond Parity: Fairness Objectives for Collaborative Filtering](https://mlsanity.com/paper/1705.08804/) (2017)
- [FairALM: Augmented Lagrangian Method for Training Fair Models with Little Regret](https://mlsanity.com/paper/2004.01355/) (2020)
- [Tensor Canonical Correlation Analysis for Multi-view Dimension Reduction](https://mlsanity.com/paper/1502.02330/) (2015)
- [Efficient Algorithms for Large-scale Generalized Eigenvector Computation and Canonical Correlation Analysis](https://mlsanity.com/paper/1604.03930/) (2016)
- [Stochastic Canonical Correlation Analysis](https://mlsanity.com/paper/1702.06533/) (2017)
- [Stochastic Approximation for Canonical Correlation Analysis](https://mlsanity.com/paper/1702.06818/) (2017)
- [Doubly Accelerated Methods for Faster CCA and Generalized Eigendecomposition](https://mlsanity.com/paper/1607.06017/) (2016)
- [Four lectures on probabilistic methods for data science](https://mlsanity.com/paper/1612.06661/) (2016)
- [Finding Linear Structure in Large Datasets with Scalable Canonical Correlation Analysis](https://mlsanity.com/paper/1506.08170/) (2015)
- [Stochastic Optimization for Deep CCA via Nonlinear Orthogonal Iterations](https://mlsanity.com/paper/1510.02054/) (2015)
- [Efficient Globally Convergent Stochastic Optimization for Canonical Correlation Analysis](https://mlsanity.com/paper/1604.01870/) (2016)
- [Deep Multi-View Learning via Task-Optimal CCA](https://mlsanity.com/paper/1907.07739/) (2019)

## Built on by (most cited first, 7 of 7)

- [On the Versatile Uses of Partial Distance Correlation in Deep Learning](https://mlsanity.com/paper/2207.09684/) (2022)
- [Optimization without Retraction on the Random Generalized Stiefel Manifold](https://mlsanity.com/paper/2405.01702/) (2024)
- [Normative framework for deriving neural networks with multi-compartmental neurons and non-Hebbian plasticity](https://mlsanity.com/paper/2302.10051/) (2023)
- [Fair Canonical Correlation Analysis](https://mlsanity.com/paper/2309.15809/) (2023)
- [The Symmetric Generalized Eigenvalue Problem as a Nash Equilibrium](https://mlsanity.com/paper/2206.04993/) (2022)
- [A Generalized EigenGame with Extensions to Multiview Representation Learning](https://mlsanity.com/paper/2211.11323/) (2022)
- [Sliding Window Informative Canonical Correlation Analysis](https://mlsanity.com/paper/2507.17921/) (2025)

## Abstract (arXiv)

We present an efficient stochastic algorithm (RSG+) for canonical correlation analysis (CCA) using a reparametrization of the projection matrices. We show how this reparametrization (into structured matrices), simple in hindsight, directly presents an opportunity to repurpose/adjust mature techniques for numerical optimization on Riemannian manifolds. Our developments nicely complement existing methods for this problem which either require O(d3) time complexity per iteration with O(frac1√t) convergence rate (where d is the dimensionality) or only extract the top 1 component with O(frac1t) convergence rate. In contrast, our algorithm offers a strict improvement for this classical problem: it achieves O(d2k) runtime complexity per iteration for extracting the top k canonical components with O(frac1t) convergence rate. While the paper primarily focuses on the formulation and technical analysis of its properties, our experiments show that the empirical behavior on common datasets is quite promising. We also explore a potential application in training fair models where the label of protected attribute is missing or otherwise unavailable.