# Institution-Specific LLM Prompting Recovers PHI That De-identification Systems and Their Gold Standards Both Miss

arXiv: https://arxiv.org/abs/2608.17051 · HTML: https://mlsanity.com/paper/2608.17051/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 49 references that are in the map.
- First posted on arXiv 17 Aug 2026.

## Builds on (most cited first, 20 of 20)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?](https://mlsanity.com/paper/2402.18272/) (2024)
- [DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4](https://mlsanity.com/paper/2303.11032/) (2023)
- [De-identification of Patient Notes with Recurrent Neural Networks](https://mlsanity.com/paper/1606.03475/) (2016)
- [Prover-Verifier Games improve legibility of LLM outputs](https://mlsanity.com/paper/2407.13692/) (2024)
- [BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP](https://mlsanity.com/paper/2506.10896/) (2025)
- [A new paradigm for accelerating clinical data science at Stanford Medicine](https://mlsanity.com/paper/2003.10534/) (2020)
- [Unlocking the Potential of Large Language Models for Clinical Text Anonymization: A Comparative Study](https://mlsanity.com/paper/2406.00062/) (2024)
- [Matching Known Patients to Health Records in Washington State Data](https://mlsanity.com/paper/1307.1370/) (2013)
- [RedactOR: An LLM-Powered Framework for Automatic Clinical Data De-Identification](https://mlsanity.com/paper/2505.18380/) (2025)
- [Not What the Doctor Ordered: Surveying LLM-based De-identification and Quantifying Clinical Information Loss](https://mlsanity.com/paper/2509.14464/) (2025)
- [Towards Automatic Evaluation and Selection of PHI De-identification Models via Multi-Agent Collaboration](https://mlsanity.com/paper/2510.16194/) (2025)
- [DeIDClinic: A Risk-Aware Pseudonymization Framework for Clinical Text De-identification and Re-identification Risk Assessment](https://mlsanity.com/paper/2410.01648/) (2024)
- [OpenMed NER: Open-Source, Domain-Adapted State-of-the-Art Transformers for Biomedical NER Across 12 Public Datasets](https://mlsanity.com/paper/2508.01630/) (2025)
- [OEMA: Ontology-Enhanced Multi-Agent Collaboration Framework for Zero-Shot Clinical Named Entity Recognition](https://mlsanity.com/paper/2511.15211/) (2025)
- [Towards Fair and Efficient De-identification: Quantifying the Efficiency and Generalizability of De-identification Approaches](https://mlsanity.com/paper/2602.15869/) (2026)

## Abstract (arXiv)

Secondary use of electronic health records requires de-identification, yet existing systems miss institutionally situated protected health information (PHI) such as hospital abbreviations, building names, and internal codes whose status is locally determined. We ask whether large language models (LLMs) with in-context learning (ICL) can close this gap and control the precision--recall trade-off. On 100 annotated pediatric oncology notes (5,322 PHI spans) from Texas Children's Hospital, we benchmarked eight LLMs against two purpose-built systems (Stanford TiDE, OpenMed PII) and two pattern-based baselines. Each LLM ran under three prompts of increasing specificity: (1) a HIPAA-aligned baseline, (2) baseline plus the institutional PHI categories it missed, and (3) prompt 2 plus instructions against over-redacting clinical content. We then compared 14~multi-agent and ensemble configurations against the best single prompt, with recall the primary safety metric. LLMs outperformed the purpose-built systems (best F1=0.918±0.001 vs. TiDE 0.779), with advantages concentrated in contextual categories. Naming the missed categories recovered 79% (48/61) of them, and discouraging over-redaction restored precision. No agentic architecture beat calibrated single-pass prompting (F1 0.906--0.907), but LLM outputs surfaced 414~candidate annotation gaps; re-annotation confirmed 227~PHI spans, against which the final prompt reached recall=0.981 (F1=0.907±0.002). Well-calibrated ICL resolves both the institutional PHI gap and the precision--recall trade-off in one LLM call per note. LLMs cost more to run than traditional methods, but that cost buys a way to audit the reference standard. LLMs are a legitimate, adaptable alternative to purpose-built de-identification systems; institution-specific prompt development should be the primary adaptation strategy.