# Correspondence Analysis and PMI-Based Word Embeddings: A Comparative Study

arXiv: https://arxiv.org/abs/2405.20895 · HTML: https://mlsanity.com/paper/2405.20895/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 83 references that are in the map.
- First posted on arXiv 31 May 2024.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [word2vec Parameter Learning Explained](https://mlsanity.com/paper/1411.2738/) (2014)
- [Trained on 100 million words and still in shape: BERT meets British National Corpus](https://mlsanity.com/paper/2303.09859/) (2023)
- [Swivel: Improving Embeddings by Noticing What's Missing](https://mlsanity.com/paper/1602.02215/) (2016)
- [LearningWord Embeddings for Low-resource Languages by PU Learning](https://mlsanity.com/paper/1805.03366/) (2018)
- [A comparison of latent semantic analysis and correspondence analysis of document-term matrices](https://mlsanity.com/paper/2108.06197/) (2021)
- [Improving information retrieval through correspondence analysis instead of latent semantic analysis](https://mlsanity.com/paper/2303.08030/) (2023)

## Built on by (most cited first, 1 of 1)

- [A Statistical Inference Framework for PMI Estimation and SGNS Word Embeddings](https://mlsanity.com/paper/2610.05058/) (2026)

## Abstract (arXiv)

Popular word embedding methods such as GloVe and Word2Vec are related to the factorization of the pointwise mutual information (PMI) matrix. In this paper, we establish a formal connection between correspondence analysis (CA) and PMI-based word embedding methods. CA is a dimensionality reduction method that uses singular value decomposition (SVD), and we show that CA is mathematically close to the weighted factorization of the PMI matrix. We further introduce variants of CA for word-context matrices, namely CA applied after a square-root transformation (ROOT-CA) and after a fourth-root transformation (ROOTROOT-CA). We analyze the performance of these methods and examine how their success or failure is influenced by extreme values in the decomposed matrix. Although our primary focus is on traditionalstatic word embedding methods, we also include a comparison with a transformer-based encoder (BERT) to situate the results relative to contextual embeddings. Empirical evaluations across multiple corpora and word-similarity benchmarks show that ROOT-CA and ROOTROOT-CA perform slightly better overall than standard PMI-based methods and achieve results competitive with BERT.