# Word Emdeddings through Hellinger PCA

arXiv: https://arxiv.org/abs/1312.5542 · HTML: https://mlsanity.com/paper/1312.5542/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2013.

## Built on by (most cited first, 10 of 10)

- [Contextual Bandits with Large Action Spaces: Made Practical](https://mlsanity.com/paper/2207.05836/) (2022)
- [ReAGent: A Model-agnostic Feature Attribution Method for Generative Language Models](https://mlsanity.com/paper/2402.00794/) (2024)
- [Conversational Agents: Theory and Applications](https://mlsanity.com/paper/2202.03164/) (2022)
- [A Survey On Neural Word Embeddings](https://mlsanity.com/paper/2110.01804/) (2021)
- [A Comprehensive Empirical Evaluation of Existing Word Embedding Approaches](https://mlsanity.com/paper/2303.07196/) (2023)
- [Revisiting PCA for time series reduction in temporal dimension](https://mlsanity.com/paper/2412.19423/) (2024)
- [EELBERT: Tiny Models through Dynamic Embeddings](https://mlsanity.com/paper/2310.20144/) (2023)
- [Word Embedding Techniques for Classification of Star Ratings](https://mlsanity.com/paper/2504.13653/) (2025)
- [Interpretable Topic Extraction and Word Embedding Learning using row-stochastic DEDICOM](https://mlsanity.com/paper/2507.16695/) (2025)
- [The Latent Diagnostic Taxonomy: A Framework for Constructing Classifiers and Diagnosing Their Decisions, Applied to Prompt Injection Detection](https://mlsanity.com/paper/2608.26423/) (2026)

## Abstract (arXiv)

Word embeddings resulting from neural language models have been shown to be successful for a large variety of NLP tasks. However, such architecture might be difficult to train and time-consuming. Instead, we propose to drastically simplify the word embeddings computation through a Hellinger PCA of the word co-occurence matrix. We compare those new word embeddings with some well-known embeddings on NER and movie review tasks and show that we can reach similar or even better performance. Although deep learning is not really necessary for generating good word embeddings, we show that it can provide an easy way to adapt embeddings to specific tasks.