# Compressing Neural Language Models by Sparse Word Representations

arXiv: https://arxiv.org/abs/1610.03950 · HTML: https://mlsanity.com/paper/1610.03950/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2016.

## Built on by (most cited first, 4 of 4)

- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Survey: Exploiting Data Redundancy for Optimization of Deep Learning](https://mlsanity.com/paper/2208.13363/) (2022)
- [Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings](https://mlsanity.com/paper/2403.14001/) (2024)
- [Optimal Size-Performance Tradeoffs: Weighing PoS Tagger Models](https://mlsanity.com/paper/2104.07951/) (2021)

## Abstract (arXiv)

Neural networks are among the state-of-the-art techniques for language modeling. Existing neural language models typically map discrete words to distributed, dense vector representations. After information processing of the preceding context words by hidden layers, an output layer estimates the probability of the next word. Such approaches are time- and memory-intensive because of the large numbers of parameters for word embeddings and the output layer. In this paper, we propose to compress neural language models by sparse word representations. In the experiments, the number of parameters in our model increases very slowly with the growth of the vocabulary size, which is almost imperceptible. Moreover, our approach not only reduces the parameter space to a large extent, but also improves the performance in terms of the perplexity measure.