# The NiuTrans Machine Translation Systems for WMT21

arXiv: https://arxiv.org/abs/2109.10485 · HTML: https://mlsanity.com/paper/2109.10485/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 22 Sep 2021.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018)
- [Ensemble Distillation for Neural Machine Translation](https://mlsanity.com/paper/1702.01802/) (2017)
- [Shallow-to-Deep Training for Neural Machine Translation](https://mlsanity.com/paper/2010.03737/) (2020)
- [Learning Light-Weight Translation Models from Deep Transformer](https://mlsanity.com/paper/2012.13866/) (2020)
- [Facebook AI's WMT20 News Translation Task Submission](https://mlsanity.com/paper/2011.08298/) (2020)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation](https://mlsanity.com/paper/2104.02308/) (2021)
- [WeChat Neural Machine Translation Systems for WMT20](https://mlsanity.com/paper/2010.00247/) (2020)
- [The Volctrans Machine Translation System for WMT20](https://mlsanity.com/paper/2010.14806/) (2020)

## Built on by (most cited first, 4 of 4)

- [The unreasonable effectiveness of few-shot learning for machine translation](https://mlsanity.com/paper/2302.01398/) (2023)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation](https://mlsanity.com/paper/2203.09176/) (2022)
- [Neural Machine Translation for the Indigenous Languages of the Americas: An Introduction](https://mlsanity.com/paper/2306.06804/) (2023)
- [Extending the Subwording Model of Multilingual Pretrained Models for New Languages](https://mlsanity.com/paper/2211.15965/) (2022)

## Abstract (arXiv)

This paper describes NiuTrans neural machine translation systems of the WMT 2021 news translation tasks. We made submissions to 9 language directions, including EnglishleftrightarrowChinese, Japanese, Russian, Icelandic and EnglishrightarrowHausa tasks. Our primary systems are built on several effective variants of Transformer, e.g., Transformer-DLCL, ODE-Transformer. We also utilize back-translation, knowledge distillation, post-ensemble, and iterative fine-tuning techniques to enhance the model performance further.