# Extremely Small BERT Models from Mixed-Vocabulary Training

arXiv: https://arxiv.org/abs/1909.11687 · HTML: https://mlsanity.com/paper/1909.11687/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2019.

## Built on by (most cited first, 8 of 8)

- [Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation](https://mlsanity.com/paper/2107.02629/) (2021)
- [KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation](https://mlsanity.com/paper/2109.06243/) (2021)
- [XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation](https://mlsanity.com/paper/2106.04563/) (2021)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models](https://mlsanity.com/paper/2411.05036/) (2024)
- [Two-Stage Fine-Tuning: A Novel Strategy for Learning Class-Imbalanced Data](https://mlsanity.com/paper/2207.10858/) (2022)
- [Extreme compression of sentence-transformer ranker models: faster inference, longer battery life, and less storage on edge devices](https://mlsanity.com/paper/2207.12852/) (2022)
- [idT5: Indonesian Version of Multilingual T5 Transformer](https://mlsanity.com/paper/2302.00856/) (2023)

## Abstract (arXiv)

Pretrained language models like BERT have achieved good results on NLP tasks, but are impractical on resource-limited devices due to memory footprint. A large fraction of this footprint comes from the input embeddings with large input vocabulary and embedding dimensions. Existing knowledge distillation methods used for model compression cannot be directly applied to train student models with reduced vocabulary sizes. To this end, we propose a distillation method to align the teacher and student embeddings via mixed-vocabulary training. Our method compresses BERT-LARGE to a task-agnostic model with smaller vocabulary and hidden dimensions, which is an order of magnitude smaller than other distilled BERT models and offers a better size-accuracy trade-off on language understanding benchmarks as well as a practical dialogue task.