# How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection

arXiv: https://arxiv.org/abs/1902.10339 · HTML: https://mlsanity.com/paper/1902.10339/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2019.

## Built on by (most cited first, 11 of 11)

- [CoDesc: A Large Code-Description Parallel Dataset](https://mlsanity.com/paper/2105.14220/) (2021)
- [Fall of Giants: How popular text-based MLaaS fall against a simple evasion attack](https://mlsanity.com/paper/2104.05996/) (2021)
- [Large Vocabulary Size Improves Large Language Models](https://mlsanity.com/paper/2406.16508/) (2024)
- [BinBert: Binary Code Understanding with a Fine-tunable and Execution-aware Transformer](https://mlsanity.com/paper/2208.06692/) (2022)
- [Power of Explanations: Towards automatic debiasing in hate speech detection](https://mlsanity.com/paper/2209.09975/) (2022)
- [Structure-informed Positional Encoding for Music Generation](https://mlsanity.com/paper/2402.13301/) (2024)
- [Explainable Health Risk Predictor with Transformer-based Medicare Claim Encoder](https://mlsanity.com/paper/2105.09428/) (2021)
- [Improving Classifier Training Efficiency for Automatic Cyberbullying Detection with Feature Density](https://mlsanity.com/paper/2111.01689/) (2021)
- [UniNet: A Unified Multi-granular Traffic Modeling Framework for Network Security](https://mlsanity.com/paper/2503.04174/) (2025)
- [Adversarially Robust and Explainable Model Compression with On-Device Personalization for Text Classification](https://mlsanity.com/paper/2101.05624/) (2021)
- [Rule by Rule: Learning with Confidence through Vocabulary Expansion](https://mlsanity.com/paper/2411.00049/) (2024)

## Abstract (arXiv)

With the rapid development in deep learning, deep neural networks have been widely adopted in many real-life natural language applications. Under deep neural networks, a pre-defined vocabulary is required to vectorize text inputs. The canonical approach to select pre-defined vocabulary is based on the word frequency, where a threshold is selected to cut off the long tail distribution. However, we observed that such simple approach could easily lead to under-sized vocabulary or over-sized vocabulary issues. Therefore, we are interested in understanding how the end-task classification accuracy is related to the vocabulary size and what is the minimum required vocabulary size to achieve a specific performance. In this paper, we provide a more sophisticated variational vocabulary dropout (VVD) based on variational dropout to perform vocabulary selection, which can intelligently select the subset of the vocabulary to achieve the required performance. To evaluate different algorithms on the newly proposed vocabulary selection problem, we propose two new metrics: Area Under Accuracy-Vocab Curve and Vocab Size under X% Accuracy Drop. Through extensive experiments on various NLP classification tasks, our variational framework is shown to significantly outperform the frequency-based and other selection baselines on these metrics.