# Slim Embedding Layers for Recurrent Neural Language Models

arXiv: https://arxiv.org/abs/1711.09873 · HTML: https://mlsanity.com/paper/1711.09873/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2017.

## Built on by (most cited first, 5 of 5)

- [KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation](https://mlsanity.com/paper/2109.06243/) (2021)
- [MorphTE: Injecting Morphology in Tensorized Embeddings](https://mlsanity.com/paper/2210.15379/) (2022)
- [From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables](https://mlsanity.com/paper/2104.08677/) (2021)
- [Efficient model compression with Random Operation Access Specific Tile (ROAST) hashing](https://mlsanity.com/paper/2207.10702/) (2022)
- [A Pipeline of Augmentation and Sequence Embedding for Classification of Imbalanced Network Traffic](https://mlsanity.com/paper/2502.18909/) (2025)

## Abstract (arXiv)

Recurrent neural language models are the state-of-the-art models for language modeling. When the vocabulary size is large, the space taken to store the model parameters becomes the bottleneck for the use of recurrent neural language models. In this paper, we introduce a simple space compression method that randomly shares the structured parameters at both the input and output embedding layers of the recurrent neural language models to significantly reduce the size of model parameters, but still compactly represent the original input and output embedding layers. The method is easy to implement and tune. Experiments on several data sets show that the new method can get similar perplexity and BLEU score results while only using a very tiny fraction of parameters.