# Neural Machine Translation of Clinical Text: An Empirical Investigation into Multilingual Pre-Trained Language Models and Transfer-Learning

arXiv: https://arxiv.org/abs/2312.07250 · HTML: https://mlsanity.com/paper/2312.07250/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 67 references that are in the map.
- First posted on arXiv 12 Dec 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [Facebook AI WMT21 News Translation Task Submission](https://mlsanity.com/paper/2108.03265/) (2021)
- [A Survey on Transfer Learning in Natural Language Processing](https://mlsanity.com/paper/2007.04239/) (2020)
- [Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods](https://mlsanity.com/paper/2105.03311/) (2021)
- [Marian: Cost-effective High-Quality Neural Machine Translation in C++](https://mlsanity.com/paper/1805.12096/) (2018)
- [HOPE: A Task-Oriented and Human-Centric Evaluation Framework Using Professional Post-Editing Towards More Effective MT Evaluation](https://mlsanity.com/paper/2112.13833/) (2021)
- [Nematus: a Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1703.04357/) (2017)
- [Chinese Character Decomposition for Neural MT with Multi-Word Expressions](https://mlsanity.com/paper/2104.04497/) (2021)
- [An Exploration of Neural Sequence-to-Sequence Architectures for Automatic Post-Editing](https://mlsanity.com/paper/1706.04138/) (2017)
- [CPAS: the UK's National Machine Learning-based Hospital Capacity Planning System for COVID-19](https://mlsanity.com/paper/2007.13825/) (2020)
- [Deployment of a Free-Text Analytics Platform at a UK National Health Service Research Hospital: CogStack at University College London Hospitals](https://mlsanity.com/paper/2108.06835/) (2021)
- [Investigating Massive Multilingual Pre-Trained Machine Translation Models for Clinical Domain via Transfer Learning](https://mlsanity.com/paper/2210.06068/) (2022)
- [Span-based Named Entity Recognition by Generating and Compressing Information](https://mlsanity.com/paper/2302.05392/) (2023)
- [Topic Modelling of Swedish Newspaper Articles about Coronavirus: a Case Study using Latent Dirichlet Allocation Method](https://mlsanity.com/paper/2301.03029/) (2023)

## Built on by (most cited first, 8 of 8)

- [The Multi-Range Theory of Translation Quality Measurement: MQM scoring models and Statistical Quality Control](https://mlsanity.com/paper/2405.16969/) (2024)
- [MaLei at MultiClinSUM: Summarisation of Clinical Documents using Perspective-Aware Iterative Self-Prompting with LLMs](https://mlsanity.com/paper/2509.07622/) (2025)
- [Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)](https://mlsanity.com/paper/2501.05465/) (2025)
- [Non-Linear Scoring Model for Translation Quality Evaluation](https://mlsanity.com/paper/2511.13467/) (2025)
- [LLMs-in-the-loop Part-1: Expert Small AI Models for Bio-Medical Text Translation](https://mlsanity.com/paper/2407.12126/) (2024)
- [Leveraging In-Context Learning and Retrieval-Augmented Generation for Automatic Question Generation in Educational Domains](https://mlsanity.com/paper/2501.17397/) (2025)
- [CANTONMT: Investigating Back-Translation and Model-Switch Mechanisms for Cantonese-English Neural Machine Translation](https://mlsanity.com/paper/2405.08172/) (2024)
- [Analyzing Cancer Patients' Experiences with Embedding-based Topic Modeling and LLMs](https://mlsanity.com/paper/2601.12154/) (2026)

## Abstract (arXiv)

We conduct investigations on clinical text machine translation by examining multilingual neural network models using deep learning such as Transformer based structures. Furthermore, to address the language resource imbalance issue, we also carry out experiments using a transfer learning methodology based on massive multilingual pre-trained language models (MMPLMs). The experimental results on three subtasks including 1) clinical case (CC), 2) clinical terminology (CT), and 3) ontological concept (OC) show that our models achieved top-level performances in the ClinSpEn-2022 shared task on English-Spanish clinical domain data. Furthermore, our expert-based human evaluations demonstrate that the small-sized pre-trained language model (PLM) won over the other two extra-large language models by a large margin, in the clinical domain fine-tuning, which finding was never reported in the field. Finally, the transfer learning method works well in our experimental setting using the WMT21fb model to accommodate a new language space Spanish that was not seen at the pre-training stage within WMT21fb itself, which deserves more exploitation for clinical knowledge transformation, e.g. to investigate into more languages. These research findings can shed some light on domain-specific machine translation development, especially in clinical and healthcare fields. Further research projects can be carried out based on our work to improve healthcare text analytics and knowledge transformation. Our data will be openly available for research purposes at https://github.com/HECTA-UoM/ClinicalNMT