# Exploring Low-Cost Transformer Model Compression for Large-Scale Commercial Reply Suggestions

arXiv: https://arxiv.org/abs/2111.13999 · HTML: https://mlsanity.com/paper/2111.13999/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 27 Nov 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [Efficient Natural Language Response Suggestion for Smart Reply](https://mlsanity.com/paper/1705.00652/) (2017)
- [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://mlsanity.com/paper/2002.12804/) (2020)
- [Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://mlsanity.com/paper/2002.08307/) (2020)
- [ConveRT: Efficient and Accurate Conversational Representations from Transformers](https://mlsanity.com/paper/1911.03688/) (2019)
- [What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning](https://mlsanity.com/paper/1911.03090/) (2019)
- [Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation](https://mlsanity.com/paper/2002.10260/) (2020)
- [Diversifying Reply Suggestions using a Matching-Conditional Variational Autoencoder](https://mlsanity.com/paper/1903.10630/) (2019)

## Abstract (arXiv)

Fine-tuning pre-trained language models improves the quality of commercial reply suggestion systems, but at the cost of unsustainable training times. Popular training time reduction approaches are resource intensive, thus we explore low-cost model compression techniques like Layer Dropping and Layer Freezing. We demonstrate the efficacy of these techniques in large-data scenarios, enabling the training time reduction for a commercial email reply suggestion system by 42%, without affecting the model relevance or user engagement. We further study the robustness of these techniques to pre-trained model and dataset size ablation, and share several insights and recommendations for commercial applications.