# Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translation

arXiv: https://arxiv.org/abs/2202.13663 · HTML: https://mlsanity.com/paper/2202.13663/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 28 Feb 2022.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [BAM! Born-Again Multi-Task Networks for Natural Language Understanding](https://mlsanity.com/paper/1907.04829/) (2019)
- [Towards Making the Most of BERT in Neural Machine Translation](https://mlsanity.com/paper/1908.05672/) (2019)
- [Selective Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2105.12967/) (2021)
- [Regularizing Neural Machine Translation by Target-bidirectional Agreement](https://mlsanity.com/paper/1808.04064/) (2018)
- [Semantic Neural Machine Translation using AMR](https://mlsanity.com/paper/1902.07282/) (2019)
- [Language Model Prior for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2004.14928/) (2020)
- [Variational Neural Machine Translation](https://mlsanity.com/paper/1605.07869/) (2016)
- [Pre-trained Language Model Representations for Language Generation](https://mlsanity.com/paper/1903.09722/) (2019)
- [Prevent the Language Model from being Overconfident in Neural Machine Translation](https://mlsanity.com/paper/2105.11098/) (2021)
- [Acquiring Knowledge from Pre-trained Model to Neural Machine Translation](https://mlsanity.com/paper/1912.01774/) (2019)
- [Synchronous Bidirectional Neural Machine Translation](https://mlsanity.com/paper/1905.04847/) (2019)
- [Variational Recurrent Neural Machine Translation](https://mlsanity.com/paper/1801.05119/) (2018)
- [Asynchronous Bidirectional Decoding for Neural Machine Translation](https://mlsanity.com/paper/1801.05122/) (2018)
- [Towards Decoding as Continuous Optimization in Neural Machine Translation](https://mlsanity.com/paper/1701.02854/) (2017)
- [DTMT: A Novel Deep Transition Architecture for Neural Machine Translation](https://mlsanity.com/paper/1812.07807/) (2018)
- [Synchronous Bidirectional Inference for Neural Sequence Generation](https://mlsanity.com/paper/1902.08955/) (2019)

## Built on by (most cited first, 7 of 7)

- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2305.08096/) (2023)
- [Towards Better Document-level Relation Extraction via Iterative Inference](https://mlsanity.com/paper/2211.14470/) (2022)
- [Efficient Modeling of Future Context for Image Captioning](https://mlsanity.com/paper/2207.10897/) (2022)
- [Neural Machine Translation with Dynamic Graph Convolutional Decoder](https://mlsanity.com/paper/2305.17698/) (2023)
- [Unified Model Learning for Various Neural Machine Translation](https://mlsanity.com/paper/2305.02777/) (2023)
- [Leveraging Diverse Modeling Contexts with Collaborating Learning for Neural Machine Translation](https://mlsanity.com/paper/2402.18428/) (2024)

## Abstract (arXiv)

Most dominant neural machine translation (NMT) models are restricted to make predictions only according to the local context of preceding words in a left-to-right manner. Although many previous studies try to incorporate global information into NMT models, there still exist limitations on how to effectively exploit bidirectional global context. In this paper, we propose a Confidence Based Bidirectional Global Context Aware (CBBGCA) training framework for NMT, where the NMT model is jointly trained with an auxiliary conditional masked language model (CMLM). The training consists of two stages: (1) multi-task joint training; (2) confidence based knowledge distillation. At the first stage, by sharing encoder parameters, the NMT model is additionally supervised by the signal from the CMLM decoder that contains bidirectional global contexts. Moreover, at the second stage, using the CMLM as teacher, we further pertinently incorporate bidirectional global context to the NMT model on its unconfidently-predicted target words via knowledge distillation. Experimental results show that our proposed CBBGCA training framework significantly improves the NMT model by +1.02, +1.30 and +0.57 BLEU scores on three large-scale translation datasets, namely WMT'14 English-to-German, WMT'19 Chinese-to-English and WMT'14 English-to-French, respectively.