# A Linearly Convergent Algorithm for Distributed Principal Component Analysis

arXiv: https://arxiv.org/abs/2101.01300 · HTML: https://mlsanity.com/paper/2101.01300/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 46 references that are in the map.
- First posted on arXiv 5 Jan 2021.

## Builds on (most cited first, 9 of 9)

- [EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization](https://mlsanity.com/paper/1404.6264/) (2014)
- [On the Convergence of Decentralized Gradient Descent](https://mlsanity.com/paper/1310.7063/) (2013)
- [NEXT: In-Network Nonconvex Optimization](https://mlsanity.com/paper/1602.00591/) (2016)
- [Cloud K-SVD: A Collaborative Dictionary Learning Algorithm for Big, Distributed Data](https://mlsanity.com/paper/1412.7839/) (2014)
- [Stochastic Optimization of PCA with Capped MSG](https://mlsanity.com/paper/1307.1674/) (2013)
- [Scaling-up Distributed Processing of Data Streams for Machine Learning](https://mlsanity.com/paper/2005.08854/) (2020)
- [Harnessing Smoothness to Accelerate Distributed Optimization](https://mlsanity.com/paper/1605.07112/) (2016)
- [Exit Time Analysis for Approximations of Gradient Descent Trajectories Around Saddle Points](https://mlsanity.com/paper/2006.01106/) (2020)
- [Distributed Stochastic Algorithms for High-rate Streaming Principal Component Analysis](https://mlsanity.com/paper/2001.01017/) (2020)

## Built on by (most cited first, 8 of 8)

- [Decentralized Riemannian Gradient Descent on the Stiefel Manifold](https://mlsanity.com/paper/2102.07091/) (2021)
- [Decentralized Optimization Over the Stiefel Manifold by an Approximate Augmented Lagrangian Function](https://mlsanity.com/paper/2112.14949/) (2021)
- [Distributed Principal Subspace Analysis for Partitioned Big Data: Algorithms, Analysis, and Implementation](https://mlsanity.com/paper/2103.06406/) (2021)
- [FAST-PCA: A Fast and Exact Algorithm for Distributed Principal Component Analysis](https://mlsanity.com/paper/2108.12373/) (2021)
- [Robust Regularized Locality Preserving Indexing for Fiedler Vector Estimation](https://mlsanity.com/paper/2107.12070/) (2021)
- [Decentralized Online Riemannian Optimization for Strongly Geodesically Convex Functions](https://mlsanity.com/paper/2607.20316/) (2026)
- [Robust Singular Values based on L1-norm PCA](https://mlsanity.com/paper/2210.12097/) (2022)
- [Rapid Grassmannian Averaging with Chebyshev Polynomials](https://mlsanity.com/paper/2410.08956/) (2024)

## Abstract (arXiv)

Principal Component Analysis (PCA) is the workhorse tool for dimensionality reduction in this era of big data. While often overlooked, the purpose of PCA is not only to reduce data dimensionality, but also to yield features that are uncorrelated. Furthermore, the ever-increasing volume of data in the modern world often requires storage of data samples across multiple machines, which precludes the use of centralized PCA algorithms. This paper focuses on the dual objective of PCA, namely, dimensionality reduction and decorrelation of features, but in a distributed setting. This requires estimating the eigenvectors of the data covariance matrix, as opposed to only estimating the subspace spanned by the eigenvectors, when data is distributed across a network of machines. Although a few distributed solutions to the PCA problem have been proposed recently, convergence guarantees and/or communications overhead of these solutions remain a concern. With an eye towards communications efficiency, this paper introduces a feedforward neural network-based one time-scale distributed PCA algorithm termed Distributed Sanger's Algorithm (DSA) that estimates the eigenvectors of the data covariance matrix when data is distributed across an undirected and arbitrarily connected network of machines. Furthermore, the proposed algorithm is shown to converge linearly to a neighborhood of the true solution. Numerical results are also provided to demonstrate the efficacy of the proposed solution.