mlsanityData to 7 Oct 2026

Robust Domain Adaptation for Pre-trained Multilingual Neural Machine Translation Models

arXiv 2210.14979 ↗First posted 26 Oct 2022cs.CL cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Recent literature has demonstrated the potential of multilingual Neural Machine Translation (mNMT) models. However, the most efficient models are not well suited to specialized industries. In these cases, internal data is scarce and expensive to find in all language pairs. Therefore, fine-tuning a mNMT model on a specialized domain is hard. In this context, we decided to focus on a new task: Domain Adaptation of a pre-trained mNMT model on a single pair of language while trying to maintain model quality on generic domain data for all language pairs. The risk of loss on generic domain and on other pairs is high. This task is key for mNMT model adoption in the industry and is at the border of many others. We propose a fine-tuning procedure for the generic mNMT that combines embeddings freezing and adversarial loss. Our experiments demonstrated that the procedure improves performances on specialized data with a minimal loss in initial performances on generic domain for all languages pairs, compared to a naive standard approach (+10.0 BLEU score on specialized data, -0.01 to -0.5 BLEU on WMT and Tatoeba datasets on the other pairs with M2M100).

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 26 Oct 2022.

Builds on

33 of 43 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2016Layer Normalizationcited by 5,579
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  11. 2016Learning without Forgettingcited by 2,465
  12. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
21 more
  1. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  2. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  3. 2017Word Translation Without Parallel Datacited by 480
  4. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  5. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  6. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  7. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  8. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  9. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  10. 2016Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanismcited by 117
  11. 2019Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlationscited by 61
  12. 2019The Missing Ingredient in Zero-Shot Neural Machine Translationcited by 59
  13. 2020A Survey of Deep Learning Techniques for Neural Machine Translationcited by 50
  14. 2020Improving Zero-Shot Translation by Disentangling Positional Informationcited by 35
  15. 2017Dynamic Data Selection for Neural Machine Translationcited by 33
  16. 2020Building a Multi-domain Neural Machine Translation Model using Knowledge Distillationcited by 9
  17. 2020Batch Clustering for Multilingual News Streamingcited by 9
  18. 2021Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translationcited by 8
  19. 2017Neural Machine Translation Training in a Multi-Domain Scenariocited by 5
  20. 2016Domain specialization: a post-training domain adaptation for Neural Machine Translationcited by 3
  21. 2019Incremental Adaptation of NMT for Professional Post-editors: A User Studycited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2023Is Robustness Transferable across Languages in Multilingual Neural Machine Translation?cited by 0
  2. 2024Defining Boundaries: The Impact of Domain Specification on Cross-Language and Cross-Domain Transfer in Machine Translationcited by 0