# WeChat Neural Machine Translation Systems for WMT21

arXiv: https://arxiv.org/abs/2108.02401 · HTML: https://mlsanity.com/paper/2108.02401/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 5 Aug 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation](https://mlsanity.com/paper/2005.03642/) (2020)
- [Bridging the Gap between Training and Inference for Neural Machine Translation](https://mlsanity.com/paper/1906.02448/) (2019)
- [Tagged Back-Translation](https://mlsanity.com/paper/1906.06442/) (2019)
- [Talking-Heads Attention](https://mlsanity.com/paper/2003.02436/) (2020)
- [On the Inference Calibration of Neural Machine Translation](https://mlsanity.com/paper/2005.00963/) (2020)
- [Very Deep Transformers for Neural Machine Translation](https://mlsanity.com/paper/2008.07772/) (2020)
- [Scheduled Sampling for Transformers](https://mlsanity.com/paper/1906.07651/) (2019)
- [Selective Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2105.12967/) (2021)
- [Accelerating Neural Transformer via an Average Attention Network](https://mlsanity.com/paper/1805.00631/) (2018)
- [Parallel Scheduled Sampling](https://mlsanity.com/paper/1906.04331/) (2019)
- [DTMT: A Novel Deep Transition Architecture for Neural Machine Translation](https://mlsanity.com/paper/1812.07807/) (2018)
- [Multi-Unit Transformers for Neural Machine Translation](https://mlsanity.com/paper/2010.10743/) (2020)
- [WeChat Neural Machine Translation Systems for WMT20](https://mlsanity.com/paper/2010.00247/) (2020)
- [Confidence-Aware Scheduled Sampling for Neural Machine Translation](https://mlsanity.com/paper/2107.10427/) (2021)

## Built on by (most cited first, 11 of 11)

- [Prompting Large Language Model for Machine Translation: A Case Study](https://mlsanity.com/paper/2301.07069/) (2023)
- [Prompting PaLM for Translation: Assessing Strategies and Performance](https://mlsanity.com/paper/2211.09102/) (2022)
- [The unreasonable effectiveness of few-shot learning for machine translation](https://mlsanity.com/paper/2302.01398/) (2023)
- [Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages](https://mlsanity.com/paper/2402.12204/) (2024)
- [Scheduled Sampling Based on Decoding Steps for Neural Machine Translation](https://mlsanity.com/paper/2108.12963/) (2021)
- [Instruction Position Matters in Sequence Generation with Large Language Models](https://mlsanity.com/paper/2308.12097/) (2023)
- [DC-MBR: Distributional Cooling for Minimum Bayesian Risk Decoding](https://mlsanity.com/paper/2212.04205/) (2022)
- [SOTASTREAM: A Streaming Approach to Machine Translation Training](https://mlsanity.com/paper/2308.07489/) (2023)
- [BJTU-WeChat's Systems for the WMT22 Chat Translation Task](https://mlsanity.com/paper/2211.15009/) (2022)
- [Summer: WeChat Neural Machine Translation Systems for the WMT22 Biomedical Translation Task](https://mlsanity.com/paper/2211.15022/) (2022)
- [Towards Multiple References Era -- Addressing Data Leakage and Limited Reference Diversity in NLG Evaluation](https://mlsanity.com/paper/2308.03131/) (2023)

## Abstract (arXiv)

This paper introduces WeChat AI's participation in WMT 2021 shared news translation task on English->Chinese, English->Japanese, Japanese->English and English->German. Our systems are based on the Transformer (Vaswani et al., 2017) with several novel and effective variants. In our experiments, we employ data filtering, large-scale synthetic data generation (i.e., back-translation, knowledge distillation, forward-translation, iterative in-domain knowledge transfer), advanced finetuning approaches, and boosted Self-BLEU based model ensemble. Our constrained systems achieve 36.9, 46.9, 27.8 and 31.3 case-sensitive BLEU scores on English->Chinese, English->Japanese, Japanese->English and English->German, respectively. The BLEU scores of English->Chinese, English->Japanese and Japanese->English are the highest among all submissions, and that of English->German is the highest among all constrained submissions.