# Probabilistic Hash Embeddings for Online Learning of Categorical Features

arXiv: https://arxiv.org/abs/2511.20893 · HTML: https://mlsanity.com/paper/2511.20893/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 51 references that are in the map.
- First posted on arXiv 25 Nov 2025.

## Builds on (most cited first, 28 of 28)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Online Learning: A Modern Introduction Using Convex Optimization](https://mlsanity.com/paper/1912.13213/) (2019)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [TabDDPM: Modelling Tabular Data with Diffusion Models](https://mlsanity.com/paper/2209.15421/) (2022)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [Deep Kalman Filters](https://mlsanity.com/paper/1511.05121/) (2015)
- [Deep Reinforcement Learning in Large Discrete Action Spaces](https://mlsanity.com/paper/1512.07679/) (2015)
- [CTAB-GAN: Effective Table Data Synthesizing](https://mlsanity.com/paper/2102.08369/) (2021)
- [TABBIE: Pretrained Representations of Tabular Data](https://mlsanity.com/paper/2105.02584/) (2021)
- [CARTE: Pretraining and Transfer for Tabular Learning](https://mlsanity.com/paper/2402.16785/) (2024)
- [Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems](https://mlsanity.com/paper/1909.02107/) (2019)
- [Hash Embeddings for Efficient Word Representations](https://mlsanity.com/paper/1709.03933/) (2017)
- [Sequential Deep Learning for Credit Risk Monitoring with Tabular Financial Data](https://mlsanity.com/paper/2012.15330/) (2020)
- [Learning to Embed Categorical Features without Embedding Tables for Recommendation](https://mlsanity.com/paper/2010.10784/) (2020)
- [TabularNet: A Neural Network Architecture for Understanding Semantic Structures of Tabular Data](https://mlsanity.com/paper/2106.03096/) (2021)
- [Detecting and Adapting to Irregular Distribution Shifts in Bayesian Online Learning](https://mlsanity.com/paper/2012.08101/) (2020)
- [Getting deep recommenders fit: Bloom embeddings for sparse binary input/output networks](https://mlsanity.com/paper/1706.03993/) (2017)
- [Unified Embedding: Battle-Tested Feature Representations for Web-Scale ML Systems](https://mlsanity.com/paper/2305.12102/) (2023)
- [Transferable Adversarial Robustness for Categorical Data via Universal Robust Embeddings](https://mlsanity.com/paper/2306.04064/) (2023)

## Built on by (most cited first, 1 of 1)

- [Explainable Knowledge Tracing via Probabilistic Embeddings and Pattern-based Reasoning](https://mlsanity.com/paper/2605.09369/) (2026)

## Abstract (arXiv)

We study streaming data with categorical features where the vocabulary of categorical feature values is changing and can even grow unboundedly over time. Feature hashing is commonly used as a pre-processing step to map these categorical values into a feature space of fixed size before learning their embeddings. While these methods have been developed and evaluated for offline or batch settings, in this paper we consider online settings. We show that deterministic embeddings are sensitive to the arrival order of categories and suffer from forgetting in online learning, leading to performance deterioration. To mitigate this issue, we propose a probabilistic hash embedding (PHE) model that treats hash embeddings as stochastic and applies Bayesian online learning to learn incrementally from data. Based on the structure of PHE, we derive a scalable inference algorithm to learn model parameters and infer/update the posteriors of hash embeddings and other latent variables. Our algorithm (i) can handle an evolving vocabulary of categorical items, (ii) is adaptive to new items without forgetting old items, (iii) is implementable with a bounded set of parameters that does not grow with the number of distinct observed values on the stream, and (iv) is invariant to the item arrival order. Experiments in classification, sequence modeling, and recommendation systems in online learning setups demonstrate the superior performance of PHE while maintaining high memory efficiency (consumes as low as 2~4 memory of a one-hot embedding table). Supplementary materials are at https://github.com/aodongli/probabilistic-hash-embeddings