# Multi-context principal component analysis

arXiv: https://arxiv.org/abs/2601.15239 · HTML: https://mlsanity.com/paper/2601.15239/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 87 references that are in the map.
- First posted on arXiv 21 Jan 2026.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [The Geometry of Culture: Analyzing Meaning through Word Embeddings](https://mlsanity.com/paper/1803.09288/) (2018)
- [Benchmarking Transcriptomics Foundation Models for Perturbation Analysis : one PCA still rules them all](https://mlsanity.com/paper/2410.13956/) (2024)
- [Group Factor Analysis](https://mlsanity.com/paper/1411.5799/) (2014)
- [Subspace power method for symmetric tensor decomposition](https://mlsanity.com/paper/1912.04007/) (2019)
- [Probabilistic Contrastive Principal Component Analysis](https://mlsanity.com/paper/2012.07977/) (2020)
- [Landscape analysis of an improved power method for tensor decomposition](https://mlsanity.com/paper/2110.15821/) (2021)
- [Stacked SVD or SVD stacked? A Random Matrix Theory perspective on data integration](https://mlsanity.com/paper/2507.22170/) (2025)
- [Contrastive independent component analysis](https://mlsanity.com/paper/2407.02357/) (2024)

## Abstract (arXiv)

Principal component analysis (PCA) is a tool to capture factors that explain variation in data. Across domains, data are now collected across multiple contexts (for example, individuals with different diseases, cells of different types, or words across texts). While the factors explaining variation in data are undoubtedly shared across subsets of contexts, no tools currently exist to systematically recover such factors. We develop multi-context principal component analysis (MCPCA), a theoretical and algorithmic framework that decomposes data into factors shared across subsets of contexts. Applied to gene expression, MCPCA reveals axes of variation shared across subsets of cancer types and an axis whose variability in tumor cells, but not mean, is associated with lung cancer progression. Applied to contextualized word embeddings from language models, MCPCA maps stages of a debate on human nature, revealing a discussion between science and fiction over decades. These axes are not found by combining data across contexts or by restricting to individual contexts. MCPCA is a principled generalization of PCA to address the challenge of understanding factors underlying data across contexts.