# Stochastic Shared Embeddings: Data-driven Regularization of Embedding Layers

arXiv: https://arxiv.org/abs/1905.10630 · HTML: https://mlsanity.com/paper/1905.10630/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2019.

## Built on by (most cited first, 9 of 9)

- [CARCA: Context and Attribute-Aware Next-Item Recommendation via Cross-Attention](https://mlsanity.com/paper/2204.06519/) (2022)
- [A social context-aware graph-based multimodal attentive learning framework for disaster content classification during emergencies: a benchmark dataset and method](https://mlsanity.com/paper/2410.08814/) (2024)
- [Improving BERT Pretraining with Syntactic Supervision](https://mlsanity.com/paper/2104.10516/) (2021)
- [UniRec: A Dual Enhancement of Uniformity and Frequency in Sequential Recommendations](https://mlsanity.com/paper/2406.18470/) (2024)
- [Proxy-based Item Representation for Attribute and Context-aware Recommendation](https://mlsanity.com/paper/2312.06145/) (2023)
- [Federated Graph-based Networks with Shared Embedding](https://mlsanity.com/paper/2210.01803/) (2022)
- [Efficient Traffic Forecasting on Large-Scale Road Network by Regularized Adaptive Graph Convolution](https://mlsanity.com/paper/2506.07179/) (2025)
- [CautionSuicide: A Deep Learning Based Approach for Detecting Suicidal Ideation in Real Time Chatbot Conversation](https://mlsanity.com/paper/2401.01023/) (2024)
- [MMbeddings: Parameter-Efficient, Low-Overfitting Probabilistic Embeddings Inspired by Nonlinear Mixed Models](https://mlsanity.com/paper/2510.22198/) (2025)

## Abstract (arXiv)

In deep neural nets, lower level embedding layers account for a large portion of the total number of parameters. Tikhonov regularization, graph-based regularization, and hard parameter sharing are approaches that introduce explicit biases into training in a hope to reduce statistical complexity. Alternatively, we propose stochastically shared embeddings (SSE), a data-driven approach to regularizing embedding layers, which stochastically transitions between embeddings during stochastic gradient descent (SGD). Because SSE integrates seamlessly with existing SGD algorithms, it can be used with only minor modifications when training large scale neural networks. We develop two versions of SSE: SSE-Graph using knowledge graphs of embeddings; SSE-SE using no prior information. We provide theoretical guarantees for our method and show its empirical effectiveness on 6 distinct tasks, from simple neural networks with one hidden layer in recommender systems, to the transformer and BERT in natural languages. We find that when used along with widely-used regularization methods such as weight decay and dropout, our proposed SSE can further reduce overfitting, which often leads to more favorable generalization results.