# LLMs-in-the-loop Part-1: Expert Small AI Models for Bio-Medical Text Translation

arXiv: https://arxiv.org/abs/2407.12126 · HTML: https://mlsanity.com/paper/2407.12126/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 54 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 13 of 13)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks](https://mlsanity.com/paper/1803.01937/) (2018)
- [Adaptive Machine Translation with Large Language Models](https://mlsanity.com/paper/2301.13294/) (2023)
- [Facebook AI WMT21 News Translation Task Submission](https://mlsanity.com/paper/2108.03265/) (2021)
- [Neural Machine Translation of Clinical Text: An Empirical Investigation into Multilingual Pre-Trained Language Models and Transfer-Learning](https://mlsanity.com/paper/2312.07250/) (2023)
- [Neural-based machine translation for medical text domain. Based on European Medicines Agency leaflet texts](https://mlsanity.com/paper/1509.08644/) (2015)

## Built on by (most cited first, 1 of 1)

- [LLMs-in-the-Loop Part 2: Expert Small AI Models for Anonymization and De-identification of PHI Across Multiple Languages](https://mlsanity.com/paper/2412.10918/) (2024)

## Abstract (arXiv)

Machine translation is indispensable in healthcare for enabling the global dissemination of medical knowledge across languages. However, complex medical terminology poses unique challenges to achieving adequate translation quality and accuracy. This study introduces a novel "LLMs-in-the-loop" approach to develop supervised neural machine translation models optimized specifically for medical texts. While large language models (LLMs) have demonstrated powerful capabilities, this research shows that small, specialized models trained on high-quality in-domain (mostly synthetic) data can outperform even vastly larger LLMs. Custom parallel corpora in six languages were compiled from scientific articles, synthetically generated clinical documents, and medical texts. Our LLMs-in-the-loop methodology employs synthetic data generation, rigorous evaluation, and agent orchestration to enhance performance. We developed small medical translation models using the MarianMT base model. We introduce a new medical translation test dataset to standardize evaluation in this domain. Assessed using BLEU, METEOR, ROUGE, and BERT scores on this test set, our MarianMT-based models outperform Google Translate, DeepL, and GPT-4-Turbo. Results demonstrate that our LLMs-in-the-loop approach, combined with fine-tuning high-quality, domain-specific data, enables specialized models to outperform general-purpose and some larger systems. This research, part of a broader series on expert small models, paves the way for future healthcare-related AI developments, including deidentification and bio-medical entity extraction models. Our study underscores the potential of tailored neural translation models and the LLMs-in-the-loop methodology to advance the field through improved data generation, evaluation, agent, and modeling techniques.