# LLMs-in-the-Loop Part 2: Expert Small AI Models for Anonymization and De-identification of PHI Across Multiple Languages

arXiv: https://arxiv.org/abs/2412.10918 · HTML: https://mlsanity.com/paper/2412.10918/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 44 references that are in the map.
- First posted on arXiv 14 Dec 2024.

## Builds on (most cited first, 9 of 9)

- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4](https://mlsanity.com/paper/2303.11032/) (2023)
- [LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis](https://mlsanity.com/paper/2310.15100/) (2023)
- [Can Large Language Models emulate an inductive Thematic Analysis of semi-structured interviews? An exploration and provocation on the limits of the approach and the model](https://mlsanity.com/paper/2305.13014/) (2023)
- [Towards Reliable Medical Question Answering: Techniques and Challenges in Mitigating Hallucinations in Language Models](https://mlsanity.com/paper/2408.13808/) (2024)
- [Discovering Latent Themes in Social Media Messaging: A Machine-in-the-Loop Approach Integrating LLMs](https://mlsanity.com/paper/2403.10707/) (2024)
- [Investigating Expert-in-the-Loop LLM Discourse Patterns for Ancient Intertextual Analysis](https://mlsanity.com/paper/2409.01882/) (2024)
- [Beyond Accuracy: Automated De-Identification of Large Real-World Clinical Text Datasets](https://mlsanity.com/paper/2312.08495/) (2023)
- [LLMs-in-the-loop Part-1: Expert Small AI Models for Bio-Medical Text Translation](https://mlsanity.com/paper/2407.12126/) (2024)

## Built on by (most cited first, 3 of 3)

- [A Survey on Current Trends and Recent Advances in Text Anonymization](https://mlsanity.com/paper/2508.21587/) (2025)
- [Not What the Doctor Ordered: Surveying LLM-based De-identification and Quantifying Clinical Information Loss](https://mlsanity.com/paper/2509.14464/) (2025)
- [PersianAnonymizer: Evaluating LLM-Labeled Training for Efficient NER-based Anonymization in Persian](https://mlsanity.com/paper/2609.00958/) (2026)

## Abstract (arXiv)

The rise of chronic diseases and pandemics like COVID-19 has emphasized the need for effective patient data processing while ensuring privacy through anonymization and de-identification of protected health information (PHI). Anonymized data facilitates research without compromising patient confidentiality. This paper introduces expert small AI models developed using the LLM-in-the-loop methodology to meet the demand for domain-specific de-identification NER models. These models overcome the privacy risks associated with large language models (LLMs) used via APIs by eliminating the need to transmit or store sensitive data. More importantly, they consistently outperform LLMs in de-identification tasks, offering superior performance and reliability. Our de-identification NER models, developed in eight languages (English, German, Italian, French, Romanian, Turkish, Spanish, and Arabic) achieved f1-micro score averages of 0.966, 0.975, 0.976, 0.970, 0.964, 0.974, 0.978, and 0.953 respectively. These results establish them as the most accurate healthcare anonymization solutions, surpassing existing small models and even general-purpose LLMs such as GPT-4o. While Part-1 of this series introduced the LLM-in-the-loop methodology for bio-medical document translation, this second paper showcases its success in developing cost-effective expert small NER models in de-identification tasks. Our findings lay the groundwork for future healthcare AI innovations, including biomedical entity and relation extraction, demonstrating the value of specialized models for domain-specific challenges.