mlsanityData to 14 Oct 2026

Investigating Massive Multilingual Pre-Trained Machine Translation Models for Clinical Domain via Transfer Learning

arXiv 2210.06068 ↗First posted 12 Oct 2022cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Massively multilingual pre-trained language models (MMPLMs) are developed in recent years demonstrating superpowers and the pre-knowledge they acquire for downstream tasks. This work investigates whether MMPLMs can be applied to clinical domain machine translation (MT) towards entirely unseen languages via transfer learning. We carry out an experimental investigation using Meta-AI's MMPLMs ``wmt21-dense-24-wide-en-X and X-en (WMT21fb)'' which were pre-trained on 7 language pairs and 14 translation directions including English to Czech, German, Hausa, Icelandic, Japanese, Russian, and Chinese, and the opposite direction. We fine-tune these MMPLMs towards English-textitSpanish language pair which textitdid not exist at all in their original pre-trained corpora both implicitly and explicitly. We prepare carefully aligned textitclinical domain data for this fine-tuning, which is different from their original mixed domain knowledge. Our experimental result shows that the fine-tuning is very successful using just 250k well-aligned in-domain EN-ES segments for three sub-task translation testings: clinical cases, clinical terms, and ontology concepts. It achieves very close evaluation scores to another MMPLM NLLB from Meta-AI, which included Spanish as a high-resource setting in the pre-training. To the best of our knowledge, this is the first work on using MMPLMs towards textitclinical domain transfer-learning NMT successfully for totally unseen languages during pre-training.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 18 of its 33 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

18 of 33 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  5. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  6. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  7. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  8. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  9. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
  10. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  11. 2016Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanismcited by 117
  12. 2021Facebook AI WMT21 News Translation Task Submissioncited by 57
6 more
  1. 2022Multi Task Learning For Zero Shot Performance Prediction of Multilingual Modelscited by 20
  2. 2018Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabularycited by 18
  3. 2021HOPE: A Task-Oriented and Human-Centric Evaluation Framework Using Professional Post-Editing Towards More Effective MT Evaluationcited by 18
  4. 2021Measuring Uncertainty in Translation Quality Evaluation (TQE)cited by 16
  5. 2021cushLEPOR: customising hLEPOR metric using Optuna for higher agreement with human judgments or pre-trained language model LaBSEcited by 8
  6. 2022Examining Large Pre-Trained Language Models for Machine Translation: What You Don't Know About Itcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 3.

3 papers.

  1. 2023Neural Machine Translation of Clinical Text: An Empirical Investigation into Multilingual Pre-Trained Language Models and Transfer-Learningcited by 8
  2. 2023MedMine: Examining Pre-trained Language Models on Medication Miningcited by 2
  3. 2023Extraction of Medication and Temporal Relation from Clinical Text using Neural Language Modelscited by 1