# On the use of BERT for Neural Machine Translation

arXiv: https://arxiv.org/abs/1909.12744 · HTML: https://mlsanity.com/paper/1909.12744/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2019.

## Built on by (most cited first, 23 of 23)

- [CURE: Code-Aware Neural Machine Translation for Automatic Program Repair](https://mlsanity.com/paper/2103.00073/) (2021)
- [From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding](https://mlsanity.com/paper/2105.07316/) (2021)
- [BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation](https://mlsanity.com/paper/2109.04588/) (2021)
- [Prevent the Language Model from being Overconfident in Neural Machine Translation](https://mlsanity.com/paper/2105.11098/) (2021)
- [Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation](https://mlsanity.com/paper/2504.06225/) (2025)
- [BERTTune: Fine-Tuning Neural Machine Translation with BERTScore](https://mlsanity.com/paper/2106.02208/) (2021)
- [Changing the Representation: Examining Language Representation for Neural Sign Language Production](https://mlsanity.com/paper/2210.06312/) (2022)
- [Implementing BERT and fine-tuned RobertA to detect AI generated news by ChatGPT](https://mlsanity.com/paper/2306.07401/) (2023)
- [Evaluating Pre-Trained Models for User Feedback Analysis in Software Engineering: A Study on Classification of App-Reviews](https://mlsanity.com/paper/2104.05861/) (2021)
- [Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation](https://mlsanity.com/paper/2203.02951/) (2022)
- [Artificial intelligence for topic modelling in Hindu philosophy: mapping themes between the Upanishads and the Bhagavad Gita](https://mlsanity.com/paper/2205.11020/) (2022)
- [Better Neural Machine Translation by Extracting Linguistic Information from BERT](https://mlsanity.com/paper/2104.02831/) (2021)
- [Investigating Neural Machine Translation for Low-Resource Languages: Using Bavarian as a Case Study](https://mlsanity.com/paper/2404.08259/) (2024)
- [BERTGEN: Multi-task Generation through BERT](https://mlsanity.com/paper/2106.03484/) (2021)
- [Checks and Strategies for Enabling Code-Switched Machine Translation](https://mlsanity.com/paper/2210.05096/) (2022)
- [An Experimental Study on Pretraining Transformers from Scratch for IR](https://mlsanity.com/paper/2301.10444/) (2023)
- [FOCUS: Forging Originality through Contrastive Use in Self-Plagiarism for Language Models](https://mlsanity.com/paper/2406.00839/) (2024)
- [From Idea to Implementation: Evaluating the Influence of Large Language Models in Software Development -- An Opinion Paper](https://mlsanity.com/paper/2503.07450/) (2025)
- [EdinSaar@WMT21: North-Germanic Low-Resource Multilingual NMT](https://mlsanity.com/paper/2109.14368/) (2021)
- [Android Security using NLP Techniques: A Review](https://mlsanity.com/paper/2107.03072/) (2021)
- [Masked Adversarial Generation for Neural Machine Translation](https://mlsanity.com/paper/2109.00417/) (2021)
- [Optimizing Deep Transformers for Chinese-Thai Low-Resource Translation](https://mlsanity.com/paper/2212.12662/) (2022)
- [Integrating Pre-trained Language Model into Neural Machine Translation](https://mlsanity.com/paper/2310.19680/) (2023)

## Abstract (arXiv)

Exploiting large pretrained models for various NMT tasks have gained a lot of visibility recently. In this work we study how BERT pretrained models could be exploited for supervised Neural Machine Translation. We compare various ways to integrate pretrained BERT model with NMT model and study the impact of the monolingual data used for BERT training on the final translation quality. We use WMT-14 English-German, IWSLT15 English-German and IWSLT14 English-Russian datasets for these experiments. In addition to standard task test set evaluation, we perform evaluation on out-of-domain test sets and noise injected test sets, in order to assess how BERT pretrained representations affect model robustness.