# Improving Neural Machine Translation by Bidirectional Training

arXiv: https://arxiv.org/abs/2109.07780 · HTML: https://mlsanity.com/paper/2109.07780/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 55 references that are in the map.
- First posted on arXiv 16 Sep 2021.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Dual Learning for Machine Translation](https://mlsanity.com/paper/1611.00179/) (2016)
- [Phrase-Based & Neural Unsupervised Machine Translation](https://mlsanity.com/paper/1804.07755/) (2018)
- [Achieving Human Parity on Automatic Chinese to English News Translation](https://mlsanity.com/paper/1803.05567/) (2018)
- [Tagged Back-Translation](https://mlsanity.com/paper/1906.06442/) (2019)
- [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://mlsanity.com/paper/1611.04798/) (2016)
- [Jointly Learning to Align and Translate with Transformer Models](https://mlsanity.com/paper/1909.02074/) (2019)
- [Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information](https://mlsanity.com/paper/2010.03142/) (2020)
- [Revisiting Low-Resource Neural Machine Translation: A Case Study](https://mlsanity.com/paper/1905.11901/) (2019)
- [Norm-Based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/2006.02014/) (2020)
- [Understanding and Improving Lexical Choice in Non-Autoregressive Translation](https://mlsanity.com/paper/2012.14583/) (2020)
- [Pretraining boosts out-of-domain robustness for pose estimation](https://mlsanity.com/paper/1909.11229/) (2019)
- [Code-switching pre-training for neural machine translation](https://mlsanity.com/paper/2009.08088/) (2020)
- [Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation](https://mlsanity.com/paper/2106.00903/) (2021)
- [Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translation](https://mlsanity.com/paper/2103.02262/) (2021)
- [Self-Attention with Cross-Lingual Position Representation](https://mlsanity.com/paper/2004.13310/) (2020)

## Built on by (most cited first, 17 of 17)

- [Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?](https://mlsanity.com/paper/2308.12898/) (2023)
- [On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translation](https://mlsanity.com/paper/2209.03316/) (2022)
- [On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation](https://mlsanity.com/paper/2110.01811/) (2021)
- [A Contrastive Cross-Channel Data Augmentation Framework for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2204.07832/) (2022)
- [Vega-MT: The JD Explore Academy Translation System for WMT22](https://mlsanity.com/paper/2209.09444/) (2022)
- [PAD-Net: An Efficient Framework for Dynamic Networks](https://mlsanity.com/paper/2211.05528/) (2022)
- [BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representation](https://mlsanity.com/paper/2204.07837/) (2022)
- [Bi-SimCut: A Simple Strategy for Boosting Neural Machine Translation](https://mlsanity.com/paper/2206.02368/) (2022)
- [Improving Neural Machine Translation by Denoising Training](https://mlsanity.com/paper/2201.07365/) (2022)
- [Prompt-Learning for Cross-Lingual Relation Extraction](https://mlsanity.com/paper/2304.10354/) (2023)
- [Choose the Final Translation from NMT and LLM hypotheses Using MBR Decoding: HW-TSC's Submission to the WMT24 General MT Shared Task](https://mlsanity.com/paper/2409.14800/) (2024)
- [OmniForce: On Human-Centered, Large Model Empowered and Cloud-Edge Collaborative AutoML System](https://mlsanity.com/paper/2303.00501/) (2023)
- [Bidirectional Awareness Induction in Autoregressive Seq2Seq Models](https://mlsanity.com/paper/2408.13959/) (2024)
- [UvA-MT's Participation in the WMT23 General Translation Shared Task](https://mlsanity.com/paper/2310.09946/) (2023)
- [HW-TSC's Submission to the CCMT 2024 Machine Translation Tasks](https://mlsanity.com/paper/2409.14842/) (2024)
- [Multilingual Neural Machine Translation System for Indic to Indic Languages](https://mlsanity.com/paper/2306.12693/) (2023)
- [Integrating Pre-trained Language Model into Neural Machine Translation](https://mlsanity.com/paper/2310.19680/) (2023)

## Abstract (arXiv)

We present a simple and effective pretraining strategy -- bidirectional training (BiT) for neural machine translation. Specifically, we bidirectionally update the model parameters at the early stage and then tune the model normally. To achieve bidirectional updating, we simply reconstruct the training samples from "srcrightarrowtgt" to "src+tgtrightarrowtgt+src" without any complicated model modifications. Notably, our approach does not increase any parameters or training steps, requiring the parallel data merely. Experimental results show that BiT pushes the SOTA neural machine translation performance across 15 translation tasks on 8 language pairs (data sizes range from 160K to 38M) significantly higher. Encouragingly, our proposed model can complement existing data manipulation strategies, i.e. back translation, data distillation, and data diversification. Extensive analyses show that our approach functions as a novel bilingual code-switcher, obtaining better bilingual alignment.