# Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain

arXiv: https://arxiv.org/abs/2404.07613 · HTML: https://mlsanity.com/paper/2404.07613/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 11 Apr 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [LinkBERT: Pretraining Language Models with Document Links](https://mlsanity.com/paper/2203.15827/) (2022)
- [Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages](https://mlsanity.com/paper/2305.12182/) (2023)
- [SciFive: a text-to-text transformer model for biomedical literature](https://mlsanity.com/paper/2106.03598/) (2021)
- [PMC-LLaMA: Towards Building Open-source Language Models for Medicine](https://mlsanity.com/paper/2304.14454/) (2023)
- [ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation](https://mlsanity.com/paper/2306.09968/) (2023)
- [T-Projection: High Quality Annotation Projection for Sequence Labeling Tasks](https://mlsanity.com/paper/2212.10548/) (2022)
- [Lessons learned from the evaluation of Spanish Language Models](https://mlsanity.com/paper/2212.08390/) (2022)

## Built on by (most cited first, 15 of 15)

- [KG4Diagnosis: A Hierarchical Multi-Agent LLM Framework with Knowledge Graph Enhancement for Medical Diagnosis](https://mlsanity.com/paper/2412.16833/) (2024)
- [Multilingual Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2411.11072/) (2024)
- [Towards User-level Private Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2502.17515/) (2025)
- [CasiMedicos-Arg: A Medical Question Answering Dataset Annotated with Explanatory Argumentative Structures](https://mlsanity.com/paper/2410.05235/) (2024)
- [Document Intelligence in the Era of Large Language Models: A Survey](https://mlsanity.com/paper/2510.13366/) (2025)
- [Leveraging Online Data to Enhance Medical Knowledge in a Small Persian Language Model](https://mlsanity.com/paper/2505.16000/) (2025)
- [MoFE: Mixture of Frozen Experts Architecture](https://mlsanity.com/paper/2503.06491/) (2025)
- [RigoChat 2: an adapted language model to Spanish using a bounded dataset and reduced hardware](https://mlsanity.com/paper/2503.08188/) (2025)
- [Pruning General Large Language Models into Customized Expert Models](https://mlsanity.com/paper/2506.02561/) (2025)
- [Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance](https://mlsanity.com/paper/2604.17650/) (2026)
- [Retrieve, Generate, Evaluate: A Case Study for Medical Paraphrases Generation with Small Language Models](https://mlsanity.com/paper/2407.16565/) (2024)
- [A Perplexity and Menger Curvature-Based Approach for Similarity Evaluation of Large Language Models](https://mlsanity.com/paper/2504.04216/) (2025)
- [Evaluating Fine-Tuned LLM Model For Medical Transcription With Small Low-Resource Languages Validated Dataset](https://mlsanity.com/paper/2603.24772/) (2026)
- [Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA](https://mlsanity.com/paper/2606.19266/) (2026)
- [Gaokerena: A Small Persian Medical Language Model Family](https://mlsanity.com/paper/2608.00932/) (2026)

## Abstract (arXiv)

Research on language technology for the development of medical applications is currently a hot topic in Natural Language Understanding and Generation. Thus, a number of large language models (LLMs) have recently been adapted to the medical domain, so that they can be used as a tool for mediating in human-AI interaction. While these LLMs display competitive performance on automated medical texts benchmarks, they have been pre-trained and evaluated with a focus on a single language (English mostly). This is particularly true of text-to-text models, which typically require large amounts of domain-specific pre-training data, often not easily accessible for many languages. In this paper, we address these shortcomings by compiling, to the best of our knowledge, the largest multilingual corpus for the medical domain in four languages, namely English, French, Italian and Spanish. This new corpus has been used to train Medical mT5, the first open-source text-to-text multilingual model for the medical domain. Additionally, we present two new evaluation benchmarks for all four languages with the aim of facilitating multilingual research in this domain. A comprehensive evaluation shows that Medical mT5 outperforms both encoders and similarly sized text-to-text models for the Spanish, French, and Italian benchmarks, while being competitive with current state-of-the-art LLMs in English.