# Distilling Knowledge from Pre-trained Language Models via Text Smoothing

arXiv: https://arxiv.org/abs/2005.03848 · HTML: https://mlsanity.com/paper/2005.03848/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2020.

## Built on by (most cited first, 2 of 2)

- [Smoothed Contrastive Learning for Unsupervised Sentence Embedding](https://mlsanity.com/paper/2109.04321/) (2021)
- [Differentially Private Model Compression](https://mlsanity.com/paper/2206.01838/) (2022)

## Abstract (arXiv)

This paper studies compressing pre-trained language models, like BERT (Devlin et al.,2019), via teacher-student knowledge distillation. Previous works usually force the student model to strictly mimic the smoothed labels predicted by the teacher BERT. As an alternative, we propose a new method for BERT distillation, i.e., asking the teacher to generate smoothed word ids, rather than labels, for teaching the student model in knowledge distillation. We call this kind of methodTextSmoothing. Practically, we use the softmax prediction of the Masked Language Model(MLM) in BERT to generate word distributions for given texts and smooth those input texts using that predicted soft word ids. We assume that both the smoothed labels and the smoothed texts can implicitly augment the input corpus, while text smoothing is intuitively more efficient since it can generate more instances in one neural network forward step.Experimental results on GLUE and SQuAD demonstrate that our solution can achieve competitive results compared with existing BERT distillation methods.