# Hash Embeddings for Efficient Word Representations

arXiv: https://arxiv.org/abs/1709.03933 · HTML: https://mlsanity.com/paper/1709.03933/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2017.

## Built on by (most cited first, 30 of 55)

- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models](https://mlsanity.com/paper/2601.07372/) (2026)
- [Scaling Expert Language Models with Unsupervised Domain Discovery](https://mlsanity.com/paper/2303.14177/) (2023)
- [ItemSage: Learning Product Embeddings for Shopping Recommendations at Pinterest](https://mlsanity.com/paper/2205.11728/) (2022)
- [Embedding in Recommender Systems: A Survey](https://mlsanity.com/paper/2310.18608/) (2023)
- [Training LLMs over Neurally Compressed Text](https://mlsanity.com/paper/2404.03626/) (2024)
- [Unified Embedding: Battle-Tested Feature Representations for Web-Scale ML Systems](https://mlsanity.com/paper/2305.12102/) (2023)
- [OmniSearchSage: Multi-Task Multi-Entity Embeddings for Pinterest Search](https://mlsanity.com/paper/2404.16260/) (2024)
- [Experimental Analysis of Large-scale Learnable Vector Storage Compression](https://mlsanity.com/paper/2311.15578/) (2023)
- [Binary Code based Hash Embedding for Web-scale Applications](https://mlsanity.com/paper/2109.02471/) (2021)
- [Embedding Compression with Hashing for Efficient Representation Learning in Large-Scale Graph](https://mlsanity.com/paper/2208.05648/) (2022)
- [T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings](https://mlsanity.com/paper/2406.19223/) (2024)
- [Bolmo: Byteifying the Next Generation of Language Models](https://mlsanity.com/paper/2512.15586/) (2025)
- [Deep Feature Embedding for Tabular Data](https://mlsanity.com/paper/2408.17162/) (2024)
- [Nimble GNN Embedding with Tensor-Train Decomposition](https://mlsanity.com/paper/2206.10581/) (2022)
- [HashFormers: Towards Vocabulary-independent Pre-trained Transformers](https://mlsanity.com/paper/2210.07904/) (2022)
- [OmniSage: Large Scale, Multi-Entity Heterogeneous Graph Representation Learning](https://mlsanity.com/paper/2504.17811/) (2025)
- [Position-based Hash Embeddings For Scaling Graph Neural Networks](https://mlsanity.com/paper/2109.00101/) (2021)
- [The Evolution of Embedding Table Optimization and Multi-Epoch Training in Pinterest Ads Conversion](https://mlsanity.com/paper/2505.05605/) (2025)
- [Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs](https://mlsanity.com/paper/2606.08347/) (2026)
- [From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables](https://mlsanity.com/paper/2104.08677/) (2021)
- [A Lower Bound of Hash Codes' Performance](https://mlsanity.com/paper/2210.05899/) (2022)
- [Clustering the Sketch: A Novel Approach to Embedding Table Compression](https://mlsanity.com/paper/2210.05974/) (2022)
- [Multi hash embeddings in spaCy](https://mlsanity.com/paper/2212.09255/) (2022)
- [Handling Large-scale Cardinality in building recommendation systems](https://mlsanity.com/paper/2401.09572/) (2024)
- [Agent-centric Information Access](https://mlsanity.com/paper/2502.19298/) (2025)
- [A Collision-Free Hot-Tier Extension for Engram-Style Conditional Memory: A Controlled Study of Training Dynamics](https://mlsanity.com/paper/2601.16531/) (2026)
- [Leviathan: Decoupling Input and Output Representations in Language Models](https://mlsanity.com/paper/2601.22040/) (2026)

## Abstract (arXiv)

We present hash embeddings, an efficient method for representing words in a continuous vector form. A hash embedding may be seen as an interpolation between a standard word embedding and a word embedding created using a random hash function (the hashing trick). In hash embeddings each token is represented by k d-dimensional embeddings vectors and one k dimensional weight vector. The final d dimensional representation of the token is the product of the two. Rather than fitting the embedding vectors for each token these are selected by the hashing trick from a shared pool of B embedding vectors. Our experiments show that hash embeddings can easily deal with huge vocabularies consisting of millions of tokens. When using a hash embedding there is no need to create a dictionary before training nor to perform any kind of vocabulary pruning after training. We show that models trained using hash embeddings exhibit at least the same level of performance as models trained using regular embeddings across a wide range of tasks. Furthermore, the number of parameters needed by such an embedding is only a fraction of what is required by a regular embedding. Since standard embeddings and embeddings constructed using the hashing trick are actually just special cases of a hash embedding, hash embeddings can be considered an extension and improvement over the existing regular embedding types.