# Acquiring Knowledge from Pre-trained Model to Neural Machine Translation

arXiv: https://arxiv.org/abs/1912.01774 · HTML: https://mlsanity.com/paper/1912.01774/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2019.

## Built on by (most cited first, 30 of 30)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation](https://mlsanity.com/paper/2109.04588/) (2021)
- [MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators](https://mlsanity.com/paper/2110.06609/) (2021)
- [Training-free Lexical Backdoor Attacks on Language Models](https://mlsanity.com/paper/2302.04116/) (2023)
- [Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders](https://mlsanity.com/paper/2104.08757/) (2021)
- [Representation Learning Beyond Linear Prediction Functions](https://mlsanity.com/paper/2105.14989/) (2021)
- [AdvPicker: Effectively Leveraging Unlabeled Data via Adversarial Discriminator for Cross-Lingual NER](https://mlsanity.com/paper/2106.02300/) (2021)
- [Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognition](https://mlsanity.com/paper/2307.10757/) (2023)
- [Understanding Multi-Turn Toxic Behaviors in Open-Domain Chatbots](https://mlsanity.com/paper/2307.09579/) (2023)
- [Understanding and Improving Sequence-to-Sequence Pretraining for Neural Machine Translation](https://mlsanity.com/paper/2203.08442/) (2022)
- [Towards General Purpose Medical AI: Continual Learning Medical Foundation Model](https://mlsanity.com/paper/2303.06580/) (2023)
- [Prompt-Learning for Short Text Classification](https://mlsanity.com/paper/2202.11345/) (2022)
- [Multilingual Translation via Grafting Pre-trained Language Models](https://mlsanity.com/paper/2109.05256/) (2021)
- [Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translation](https://mlsanity.com/paper/2202.13663/) (2022)
- [Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation and Understanding](https://mlsanity.com/paper/2204.07834/) (2022)
- [Better Neural Machine Translation by Extracting Linguistic Information from BERT](https://mlsanity.com/paper/2104.02831/) (2021)
- [The Effect of Normalization for Bi-directional Amharic-English Neural Machine Translation](https://mlsanity.com/paper/2210.15224/) (2022)
- [Towards a Holistic Understanding of Mathematical Questions with Contrastive Pre-training](https://mlsanity.com/paper/2301.07558/) (2023)
- [Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusters](https://mlsanity.com/paper/2205.09470/) (2022)
- [Towards Accurate Translation via Semantically Appropriate Application of Lexical Constraints](https://mlsanity.com/paper/2306.12089/) (2023)
- [JobFormer: Skill-Aware Job Recommendation with Semantic-Enhanced Transformer](https://mlsanity.com/paper/2404.04313/) (2024)
- [ProKD: An Unsupervised Prototypical Knowledge Distillation Network for Zero-Resource Cross-Lingual Named Entity Recognition](https://mlsanity.com/paper/2301.08855/) (2023)
- [Enhanced Simultaneous Machine Translation with Word-level Policies](https://mlsanity.com/paper/2310.16417/) (2023)
- [KD4MT: A Survey of Knowledge Distillation for Machine Translation](https://mlsanity.com/paper/2602.15845/) (2026)
- [Improving Similar Language Translation With Transfer Learning](https://mlsanity.com/paper/2108.03533/) (2021)
- [Enhancing Low-Resource NMT with a Multilingual Encoder and Knowledge Distillation: A Case Study](https://mlsanity.com/paper/2407.06538/) (2024)
- [Tgea: An error-annotated dataset and benchmark tasks for text generation from pretrained language models](https://mlsanity.com/paper/2503.04232/) (2025)
- [Integrating Pre-trained Language Model into Neural Machine Translation](https://mlsanity.com/paper/2310.19680/) (2023)
- [Explainable AI in Diagnosing and Anticipating Leukemia Using Transfer Learning Method](https://mlsanity.com/paper/2312.00487/) (2023)
- [Leveraging Diverse Modeling Contexts with Collaborating Learning for Neural Machine Translation](https://mlsanity.com/paper/2402.18428/) (2024)

## Abstract (arXiv)

Pre-training and fine-tuning have achieved great success in the natural language process field. The standard paradigm of exploiting them includes two steps: first, pre-training a model, e.g. BERT, with a large scale unlabeled monolingual data. Then, fine-tuning the pre-trained model with labeled data from downstream tasks. However, in neural machine translation (NMT), we address the problem that the training objective of the bilingual task is far different from the monolingual pre-trained model. This gap leads that only using fine-tuning in NMT can not fully utilize prior language knowledge. In this paper, we propose an APT framework for acquiring knowledge from the pre-trained model to NMT. The proposed approach includes two modules: 1). a dynamic fusion mechanism to fuse task-specific features adapted from general knowledge into NMT network, 2). a knowledge distillation paradigm to learn language knowledge continuously during the NMT training process. The proposed approach could integrate suitable knowledge from pre-trained models to improve the NMT. Experimental results on WMT English to German, German to English and Chinese to English machine translation tasks show that our model outperforms strong baselines and the fine-tuning counterparts.