# Investigating Massive Multilingual Pre-Trained Machine Translation Models for Clinical Domain via Transfer Learning

arXiv: https://arxiv.org/abs/2210.06068 · HTML: https://mlsanity.com/paper/2210.06068/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://mlsanity.com/paper/2010.12858/) (2020)
- [Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism](https://mlsanity.com/paper/1601.01073/) (2016)
- [Facebook AI WMT21 News Translation Task Submission](https://mlsanity.com/paper/2108.03265/) (2021)
- [Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models](https://mlsanity.com/paper/2205.06130/) (2022)
- [Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary](https://mlsanity.com/paper/1811.01137/) (2018)
- [HOPE: A Task-Oriented and Human-Centric Evaluation Framework Using Professional Post-Editing Towards More Effective MT Evaluation](https://mlsanity.com/paper/2112.13833/) (2021)
- [Measuring Uncertainty in Translation Quality Evaluation (TQE)](https://mlsanity.com/paper/2111.07699/) (2021)
- [cushLEPOR: customising hLEPOR metric using Optuna for higher agreement with human judgments or pre-trained language model LaBSE](https://mlsanity.com/paper/2108.09484/) (2021)
- [Examining Large Pre-Trained Language Models for Machine Translation: What You Don't Know About It](https://mlsanity.com/paper/2209.07417/) (2022)

## Built on by (most cited first, 3 of 3)

- [Neural Machine Translation of Clinical Text: An Empirical Investigation into Multilingual Pre-Trained Language Models and Transfer-Learning](https://mlsanity.com/paper/2312.07250/) (2023)
- [MedMine: Examining Pre-trained Language Models on Medication Mining](https://mlsanity.com/paper/2308.03629/) (2023)
- [Extraction of Medication and Temporal Relation from Clinical Text using Neural Language Models](https://mlsanity.com/paper/2310.02229/) (2023)

## Abstract (arXiv)

Massively multilingual pre-trained language models (MMPLMs) are developed in recent years demonstrating superpowers and the pre-knowledge they acquire for downstream tasks. This work investigates whether MMPLMs can be applied to clinical domain machine translation (MT) towards entirely unseen languages via transfer learning. We carry out an experimental investigation using Meta-AI's MMPLMs ``wmt21-dense-24-wide-en-X and X-en (WMT21fb)'' which were pre-trained on 7 language pairs and 14 translation directions including English to Czech, German, Hausa, Icelandic, Japanese, Russian, and Chinese, and the opposite direction. We fine-tune these MMPLMs towards English-textitSpanish language pair which textitdid not exist at all in their original pre-trained corpora both implicitly and explicitly. We prepare carefully aligned textitclinical domain data for this fine-tuning, which is different from their original mixed domain knowledge. Our experimental result shows that the fine-tuning is very successful using just 250k well-aligned in-domain EN-ES segments for three sub-task translation testings: clinical cases, clinical terms, and ontology concepts. It achieves very close evaluation scores to another MMPLM NLLB from Meta-AI, which included Spanish as a high-resource setting in the pre-training. To the best of our knowledge, this is the first work on using MMPLMs towards textitclinical domain transfer-learning NMT successfully for totally unseen languages during pre-training.