# Distilling Knowledge Learned in BERT for Text Generation

arXiv: https://arxiv.org/abs/1911.03829 · HTML: https://mlsanity.com/paper/1911.03829/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2019.

## Built on by (most cited first, 5 of 5)

- [Reward Optimization for Neural Machine Translation with Learned Metrics](https://mlsanity.com/paper/2104.07541/) (2021)
- [Cross-Lingual Text Classification of Transliterated Hindi and Malayalam](https://mlsanity.com/paper/2108.13620/) (2021)
- [Location-Aware Visual Question Generation with Lightweight Models](https://mlsanity.com/paper/2310.15129/) (2023)
- [PAEG: Phrase-level Adversarial Example Generation for Neural Machine Translation](https://mlsanity.com/paper/2201.02009/) (2022)
- [Global Structure-Aware Drum Transcription Based on Self-Attention Mechanisms](https://mlsanity.com/paper/2105.05791/) (2021)

## Abstract (arXiv)

Large-scale pre-trained language model such as BERT has achieved great success in language understanding tasks. However, it remains an open question how to utilize BERT for language generation. In this paper, we present a novel approach, Conditional Masked Language Modeling (C-MLM), to enable the finetuning of BERT on target generation tasks. The finetuned BERT (teacher) is exploited as extra supervision to improve conventional Seq2Seq models (student) for better text generation performance. By leveraging BERT's idiosyncratic bidirectional nature, distilling knowledge learned in BERT can encourage auto-regressive Seq2Seq models to plan ahead, imposing global sequence-level supervision for coherent text generation. Experiments show that the proposed approach significantly outperforms strong Transformer baselines on multiple language generation tasks such as machine translation and text summarization. Our proposed model also achieves new state of the art on IWSLT German-English and English-Vietnamese MT datasets. Code is available at https://github.com/ChenRocks/Distill-BERT-Textgen.