# Embedding Compression for Text Classification Using Dictionary Screening

arXiv: https://arxiv.org/abs/2211.12715 · HTML: https://mlsanity.com/paper/2211.12715/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 59 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures](https://mlsanity.com/paper/1607.03250/) (2016)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [Recurrent Neural Network for Text Classification with Multi-Task Learning](https://mlsanity.com/paper/1605.05101/) (2016)
- [SGM: Sequence Generation Model for Multi-label Classification](https://mlsanity.com/paper/1806.04822/) (2018)
- [Revisiting the Nystrom Method for Improved Large-Scale Machine Learning](https://mlsanity.com/paper/1303.1849/) (2013)
- [Simple and Effective Dimensionality Reduction for Word Embeddings](https://mlsanity.com/paper/1708.03629/) (2017)
- [An Entropy-based Pruning Method for CNN Compression](https://mlsanity.com/paper/1706.05791/) (2017)
- [Compressing Word Embeddings via Deep Compositional Code Learning](https://mlsanity.com/paper/1711.01068/) (2017)
- [The unreasonable effectiveness of the forget gate](https://mlsanity.com/paper/1804.04849/) (2018)
- [Alternating Multi-bit Quantization for Recurrent Neural Networks](https://mlsanity.com/paper/1802.00150/) (2018)
- [Compressing Recurrent Neural Network with Tensor Train](https://mlsanity.com/paper/1705.08052/) (2017)
- [Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function](https://mlsanity.com/paper/2009.04007/) (2020)
- [Online Embedding Compression for Text Classification using Low Rank Matrix Factorization](https://mlsanity.com/paper/1811.00641/) (2018)
- [GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Training](https://mlsanity.com/paper/1811.03617/) (2018)
- [Grow and Prune Compact, Fast, and Accurate LSTMs](https://mlsanity.com/paper/1805.11797/) (2018)
- [Self-Adaptive Hierarchical Sentence Model](https://mlsanity.com/paper/1504.05070/) (2015)
- [On the Compression of Recurrent Neural Networks with an Application to LVCSR acoustic modeling for Embedded Speech Recognition](https://mlsanity.com/paper/1603.08042/) (2016)
- [A novel channel pruning method for deep neural network compression](https://mlsanity.com/paper/1805.11394/) (2018)
- [Compression of Recurrent Neural Networks for Efficient Language Modeling](https://mlsanity.com/paper/1902.02380/) (2019)

## Abstract (arXiv)

In this paper, we propose a dictionary screening method for embedding compression in text classification tasks. The key purpose of this method is to evaluate the importance of each keyword in the dictionary. To this end, we first train a pre-specified recurrent neural network-based model using a full dictionary. This leads to a benchmark model, which we then use to obtain the predicted class probabilities for each sample in a dataset. Next, to evaluate the impact of each keyword in affecting the predicted class probabilities, we develop a novel method for assessing the importance of each keyword in a dictionary. Consequently, each keyword can be screened, and only the most important keywords are reserved. With these screened keywords, a new dictionary with a considerably reduced size can be constructed. Accordingly, the original text sequence can be substantially compressed. The proposed method leads to significant reductions in terms of parameters, average text sequence, and dictionary size. Meanwhile, the prediction power remains very competitive compared to the benchmark model. Extensive numerical studies are presented to demonstrate the empirical performance of the proposed method.