# Better Neural Machine Translation by Extracting Linguistic Information from BERT

arXiv: https://arxiv.org/abs/2104.02831 · HTML: https://mlsanity.com/paper/2104.02831/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 7 Apr 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018)
- [Assessing BERT's Syntactic Abilities](https://mlsanity.com/paper/1901.05287/) (2019)
- [Towards Making the Most of BERT in Neural Machine Translation](https://mlsanity.com/paper/1908.05672/) (2019)
- [Linguistic Input Features Improve Neural Machine Translation](https://mlsanity.com/paper/1606.02892/) (2016)
- [Acquiring Knowledge from Pre-trained Model to Neural Machine Translation](https://mlsanity.com/paper/1912.01774/) (2019)
- [On the use of BERT for Neural Machine Translation](https://mlsanity.com/paper/1909.12744/) (2019)
- [Syntax-Infused Transformer and BERT models for Machine Translation and Natural Language Understanding](https://mlsanity.com/paper/1911.06156/) (2019)
- [Stack-propagation: Improved Representation Learning for Syntax](https://mlsanity.com/paper/1603.06598/) (2016)

## Built on by (most cited first, 5 of 5)

- [TransFool: An Adversarial Attack against Neural Machine Translation Models](https://mlsanity.com/paper/2302.00944/) (2023)
- [SpEL: Structured Prediction for Entity Linking](https://mlsanity.com/paper/2310.14684/) (2023)
- [Artificial intelligence for topic modelling in Hindu philosophy: mapping themes between the Upanishads and the Bhagavad Gita](https://mlsanity.com/paper/2205.11020/) (2022)
- [Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation](https://mlsanity.com/paper/2204.05953/) (2022)
- [A Data Selection Approach for Enhancing Low Resource Machine Translation Using Cross-Lingual Sentence Representations](https://mlsanity.com/paper/2409.02712/) (2024)

## Abstract (arXiv)

Adding linguistic information (syntax or semantics) to neural machine translation (NMT) has mostly focused on using point estimates from pre-trained models. Directly using the capacity of massive pre-trained contextual word embedding models such as BERT (Devlin et al., 2019) has been marginally useful in NMT because effective fine-tuning is difficult to obtain for NMT without making training brittle and unreliable. We augment NMT by extracting dense fine-tuned vector-based linguistic information from BERT instead of using point estimates. Experimental results show that our method of incorporating linguistic information helps NMT to generalize better in a variety of training contexts and is no more difficult to train than conventional Transformer-based NMT.