# On the Dimensionality of Embeddings for Sparse Features and Data

arXiv: https://arxiv.org/abs/1901.02103 · HTML: https://mlsanity.com/paper/1901.02103/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jan 2019.

## Built on by (most cited first, 3 of 3)

- [Enhancing Embedding Representation Stability in Recommendation Systems with Semantic ID](https://mlsanity.com/paper/2504.02137/) (2025)
- [The Ubiquitous Sparse Matrix-Matrix Products](https://mlsanity.com/paper/2508.04077/) (2025)
- [Transformer-Based Active Learning for Data-Efficient Vaccine Epitope Selection in PRRS](https://mlsanity.com/paper/2606.28659/) (2026)

## Abstract (arXiv)

In this note we discuss a common misconception, namely that embeddings are always used to reduce the dimensionality of the item space. We show that when we measure dimensionality in terms of information entropy then the embedding of sparse probability distributions, that can be used to represent sparse features or data, may or not reduce the dimensionality of the item space. However, the embeddings do provide a different and often more meaningful representation of the items for a particular task at hand. Also, we give upper bounds and more precise guidelines for choosing the embedding dimension.