# Facebook AI WMT21 News Translation Task Submission

arXiv: https://arxiv.org/abs/2108.03265 · HTML: https://mlsanity.com/paper/2108.03265/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 40 references that are in the map.
- First posted on arXiv 6 Aug 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Facebook FAIR's WMT19 News Translation Task Submission](https://mlsanity.com/paper/1907.06616/) (2019)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://mlsanity.com/paper/1911.06154/) (2019)
- [Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism](https://mlsanity.com/paper/1601.01073/) (2016)
- [Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings](https://mlsanity.com/paper/1811.01136/) (2018)
- [Cross-lingual Retrieval for Iterative Self-Supervised Training](https://mlsanity.com/paper/2006.09526/) (2020)
- [The Effect of Translationese in Machine Translation Test Sets](https://mlsanity.com/paper/1906.08069/) (2019)
- [Multilingual Neural Machine Translation with Language Clustering](https://mlsanity.com/paper/1908.09324/) (2019)
- [On The Evaluation of Machine Translation Systems Trained With Back-Translation](https://mlsanity.com/paper/1908.05204/) (2019)
- [Translationese in Machine Translation Evaluation](https://mlsanity.com/paper/1906.09833/) (2019)
- [Simple and Effective Noisy Channel Modeling for Neural Machine Translation](https://mlsanity.com/paper/1908.05731/) (2019)
- [Domain, Translationese and Noise in Synthetic Data for Neural Machine Translation](https://mlsanity.com/paper/1911.03362/) (2019)
- [Language Models not just for Pre-training: Fast Online Neural Noisy Channel Modeling](https://mlsanity.com/paper/2011.07164/) (2020)
- [Facebook AI's WAT19 Myanmar-English Translation Task Submission](https://mlsanity.com/paper/1910.06848/) (2019)

## Built on by (most cited first, 30 of 57)

- [Prompting PaLM for Translation: Assessing Strategies and Performance](https://mlsanity.com/paper/2211.09102/) (2022)
- [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://mlsanity.com/paper/2111.09388/) (2021)
- [A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation](https://mlsanity.com/paper/2205.02022/) (2022)
- [Survey of Low-Resource Machine Translation](https://mlsanity.com/paper/2109.00486/) (2021)
- [The unreasonable effectiveness of few-shot learning for machine translation](https://mlsanity.com/paper/2302.01398/) (2023)
- [Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation](https://mlsanity.com/paper/2208.05309/) (2022)
- [Do GPTs Produce Less Literal Translations?](https://mlsanity.com/paper/2305.16806/) (2023)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation](https://mlsanity.com/paper/2203.09176/) (2022)
- [Iterative Translation Refinement with Large Language Models](https://mlsanity.com/paper/2306.03856/) (2023)
- [Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment](https://mlsanity.com/paper/2404.01054/) (2024)
- [Augmenting Large Language Model Translators via Translation Memories](https://mlsanity.com/paper/2305.17367/) (2023)
- [Cedille: A large autoregressive French language model](https://mlsanity.com/paper/2202.03371/) (2022)
- [Tricks for Training Sparse Translation Models](https://mlsanity.com/paper/2110.08246/) (2021)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [Hyperparameter-Free Approach for Faster Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2401.02749/) (2024)
- [Alternative Input Signals Ease Transfer in Multilingual Machine Translation](https://mlsanity.com/paper/2110.07804/) (2021)
- [Efficient yet Competitive Speech Translation: FBK@IWSLT2022](https://mlsanity.com/paper/2205.02629/) (2022)
- [Consistent Human Evaluation of Machine Translation across Language Pairs](https://mlsanity.com/paper/2205.08533/) (2022)
- [Revisiting Checkpoint Averaging for Neural Machine Translation](https://mlsanity.com/paper/2210.11803/) (2022)
- [Learning Language-Specific Layers for Multilingual Machine Translation](https://mlsanity.com/paper/2305.02665/) (2023)
- [Domain-Specific Text Generation for Machine Translation](https://mlsanity.com/paper/2208.05909/) (2022)
- [Discord Questions: A Computational Approach To Diversity Analysis in News Coverage](https://mlsanity.com/paper/2211.05007/) (2022)
- [A Call for Clarity in Beam Search: How It Works and When It Stops](https://mlsanity.com/paper/2204.05424/) (2022)
- [Target-Agnostic Gender-Aware Contrastive Learning for Mitigating Bias in Multilingual Machine Translation](https://mlsanity.com/paper/2305.14016/) (2023)
- [Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?](https://mlsanity.com/paper/2406.16316/) (2024)
- [Vega-MT: The JD Explore Academy Translation System for WMT22](https://mlsanity.com/paper/2209.09444/) (2022)
- [Neural Machine Translation of Clinical Text: An Empirical Investigation into Multilingual Pre-Trained Language Models and Transfer-Learning](https://mlsanity.com/paper/2312.07250/) (2023)
- [The first neural machine translation system for the Erzya language](https://mlsanity.com/paper/2209.09368/) (2022)
- [Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation](https://mlsanity.com/paper/2307.09701/) (2023)
- [Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs](https://mlsanity.com/paper/2403.08845/) (2024)

## Abstract (arXiv)

We describe Facebook's multilingual model submission to the WMT2021 shared task on news translation. We participate in 14 language directions: English to and from Czech, German, Hausa, Icelandic, Japanese, Russian, and Chinese. To develop systems covering all these directions, we focus on multilingual models. We utilize data from all available sources --- WMT, large-scale data mining, and in-domain backtranslation --- to create high quality bilingual and multilingual baselines. Subsequently, we investigate strategies for scaling multilingual model size, such that one system has sufficient capacity for high quality representations of all eight languages. Our final submission is an ensemble of dense and sparse Mixture-of-Expert multilingual translation models, followed by finetuning on in-domain news data and noisy channel reranking. Compared to previous year's winning submissions, our multilingual system improved the translation quality on all language directions, with an average improvement of 2.0 BLEU. In the WMT2021 task, our system ranks first in 10 directions based on automatic evaluation.