# Can Zero-Shot Commercial APIs Deliver Regulatory-Grade Clinical Text DeIdentification?

arXiv: https://arxiv.org/abs/2503.20794 · HTML: https://mlsanity.com/paper/2503.20794/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 16 references that are in the map.
- First posted on arXiv 21 Mar 2025.

## Builds on (most cited first, 4 of 4)

- [DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4](https://mlsanity.com/paper/2303.11032/) (2023)
- [Spark NLP: Natural Language Understanding at Scale](https://mlsanity.com/paper/2101.10848/) (2021)
- [De-identification is not enough: a comparison between de-identified and synthetic clinical notes](https://mlsanity.com/paper/2402.00179/) (2024)
- [A Deep Learning Architecture for De-identification of Patient Notes: Implementation and Evaluation](https://mlsanity.com/paper/1810.01570/) (2018)

## Built on by (most cited first, 1 of 1)

- [RedactOR: An LLM-Powered Framework for Automatic Clinical Data De-Identification](https://mlsanity.com/paper/2505.18380/) (2025)

## Abstract (arXiv)

We evaluate the performance of four leading solutions for de-identification of unstructured medical text - Azure Health Data Services, AWS Comprehend Medical, OpenAI GPT-4o, and John Snow Labs - on a ground truth dataset of 48 clinical documents annotated by medical experts. The analysis, conducted at both entity-level and token-level, suggests that John Snow Labs' Medical Language Models solution achieves the highest accuracy, with a 96% F1-score in protected health information (PHI) detection, outperforming Azure (91%), AWS (83%), and GPT-4o (79%). John Snow Labs is not only the only solution which achieves regulatory-grade accuracy (surpassing that of human experts) but is also the most cost-effective solution: It is over 80% cheaper compared to Azure and GPT-4o, and is the only solution not priced by token. Its fixed-cost local deployment model avoids the escalating per-request fees of cloud-based services, making it a scalable and economical choice.