# Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task

arXiv: https://arxiv.org/abs/2111.02086 · HTML: https://mlsanity.com/paper/2111.02086/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 3 Nov 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://mlsanity.com/paper/1911.06154/) (2019)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [Learning with Feature-Dependent Label Noise: A Progressive Approach](https://mlsanity.com/paper/2103.07756/) (2021)
- [Multilingual Neural Machine Translation with Knowledge Distillation](https://mlsanity.com/paper/1902.10461/) (2019)
- [Balancing Training for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2004.06748/) (2020)
- [DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders](https://mlsanity.com/paper/2106.13736/) (2021)
- [Learning Language Specific Sub-network for Multilingual Machine Translation](https://mlsanity.com/paper/2105.09259/) (2021)
- [Multi-task Learning for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2010.02523/) (2020)
- [Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languages](https://mlsanity.com/paper/1909.09524/) (2019)
- [Shallow-to-Deep Training for Neural Machine Translation](https://mlsanity.com/paper/2010.03737/) (2020)
- [Dynamic Data Selection and Weighting for Iterative Back-Translation](https://mlsanity.com/paper/2004.03672/) (2020)
- [Adaptive Sparse Transformer for Multilingual Translation](https://mlsanity.com/paper/2104.07358/) (2021)

## Built on by (most cited first, 30 of 34)

- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models](https://mlsanity.com/paper/2411.04905/) (2024)
- [A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation](https://mlsanity.com/paper/2205.02022/) (2022)
- [SCM: Enhancing Large Language Model with Self-Controlled Memory Framework](https://mlsanity.com/paper/2304.13343/) (2023)
- [C-ICL: Contrastive In-context Learning for Information Extraction](https://mlsanity.com/paper/2402.11254/) (2024)
- [GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator](https://mlsanity.com/paper/2212.10218/) (2022)
- [P2P: Automated Paper-to-Poster Generation and Fine-Grained Benchmark](https://mlsanity.com/paper/2505.17104/) (2025)
- [Lemur: Log Parsing with Entropy Sampling and Chain-of-Thought Merging](https://mlsanity.com/paper/2402.18205/) (2024)
- [mCL-NER: Cross-Lingual Named Entity Recognition via Multi-view Contrastive Learning](https://mlsanity.com/paper/2308.09073/) (2023)
- [Tencent's Multilingual Machine Translation System for WMT22 Large-Scale African Languages](https://mlsanity.com/paper/2210.09644/) (2022)
- [Learning Language-Specific Layers for Multilingual Machine Translation](https://mlsanity.com/paper/2305.02665/) (2023)
- [FinPT: Financial Risk Prediction with Profile Tuning on Pretrained Foundation Models](https://mlsanity.com/paper/2308.00065/) (2023)
- [GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation](https://mlsanity.com/paper/2207.14467/) (2022)
- [Towards Higher Pareto Frontier in Multilingual Machine Translation](https://mlsanity.com/paper/2305.15718/) (2023)
- [M2C: Towards Automatic Multimodal Manga Complement](https://mlsanity.com/paper/2310.17130/) (2023)
- [Adaptive Neural Ranking Framework: Toward Maximized Business Goal for Cascade Ranking Systems](https://mlsanity.com/paper/2310.10462/) (2023)
- [Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategies](https://mlsanity.com/paper/2402.09282/) (2024)
- [New Intent Discovery with Attracting and Dispersing Prototype](https://mlsanity.com/paper/2403.16913/) (2024)
- [m3P: Towards Multimodal Multilingual Translation with Multimodal Prompt](https://mlsanity.com/paper/2403.17556/) (2024)
- [Building Multilingual Machine Translation Systems That Serve Arbitrary X-Y Translations](https://mlsanity.com/paper/2206.14982/) (2022)
- [Learning an Artificial Language for Knowledge-Sharing in Multilingual Translation](https://mlsanity.com/paper/2211.01292/) (2022)
- [HanoiT: Enhancing Context-aware Translation via Selective Context](https://mlsanity.com/paper/2301.06825/) (2023)
- [The YiTrans End-to-End Speech Translation System for IWSLT 2022 Offline Shared Task](https://mlsanity.com/paper/2206.05777/) (2022)
- [Tackling Low-Resourced Sign Language Translation: UPC at WMT-SLT 22](https://mlsanity.com/paper/2212.01140/) (2022)
- [Multilingual Entity and Relation Extraction from Unified to Language-specific Training](https://mlsanity.com/paper/2301.04434/) (2023)
- [Advancing Multilingual Pre-training: TRIP Triangular Document-level Pre-training for Multilingual Language Models](https://mlsanity.com/paper/2212.07752/) (2022)
- [RoNID: New Intent Discovery with Generated-Reliable Labels and Cluster-friendly Representations](https://mlsanity.com/paper/2404.08977/) (2024)
- [Towards Real-world Scenario: Imbalanced New Intent Discovery](https://mlsanity.com/paper/2406.03127/) (2024)
- [Fast Vocabulary Projection Method via Clustering for Multilingual Machine Translation on GPU](https://mlsanity.com/paper/2208.06874/) (2022)
- [Low-Resource Machine Translation through the Lens of Personalized Federated Learning](https://mlsanity.com/paper/2406.12564/) (2024)

## Abstract (arXiv)

This report describes Microsoft's machine translation systems for the WMT21 shared task on large-scale multilingual machine translation. We participated in all three evaluation tracks including Large Track and two Small Tracks where the former one is unconstrained and the latter two are fully constrained. Our model submissions to the shared task were initialized with DeltaLMfootnoteurlhttps://aka.ms/deltalm, a generic pre-trained multilingual encoder-decoder model, and fine-tuned correspondingly with the vast collected parallel data and allowed data sources according to track settings, together with applying progressive learning and iterative back-translation approaches to further improve the performance. Our final submissions ranked first on three tracks in terms of the automatic evaluation metric.