# SHIELD: A Diverse Clinical Note Dataset and Distilled Small Language Models for Enterprise-Scale De-identification

arXiv: https://arxiv.org/abs/2605.03301 · HTML: https://mlsanity.com/paper/2605.03301/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 43 references that are in the map.
- First posted on arXiv 5 May 2026.

## Builds on (most cited first, 18 of 18)

- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Small Language Models are the Future of Agentic AI](https://mlsanity.com/paper/2506.02153/) (2025)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [Effectively Unbiased FID and Inception Score and where to find them](https://mlsanity.com/paper/1911.07023/) (2019)
- [DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4](https://mlsanity.com/paper/2303.11032/) (2023)
- [The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes](https://mlsanity.com/paper/2601.11659/) (2026)
- [BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP](https://mlsanity.com/paper/2506.10896/) (2025)
- [A new paradigm for accelerating clinical data science at Stanford Medicine](https://mlsanity.com/paper/2003.10534/) (2020)
- [De-identification of clinical free text using natural language processing: A systematic review of current approaches](https://mlsanity.com/paper/2312.03736/) (2023)
- [ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performance](https://mlsanity.com/paper/2504.08716/) (2025)
- [A Pragmatic Note on Evaluating Generative Models with Fr'echet Inception Distance for Retinal Image Synthesis](https://mlsanity.com/paper/2502.17160/) (2025)
- [Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training](https://mlsanity.com/paper/2508.15390/) (2025)
- [Fr'echet Distance for Offline Evaluation of Information Retrieval Systems with Sparse Labels](https://mlsanity.com/paper/2401.17543/) (2024)
- [RedactOR: An LLM-Powered Framework for Automatic Clinical Data De-Identification](https://mlsanity.com/paper/2505.18380/) (2025)
- [Validating transformers for redaction of text from electronic health records in real-world healthcare](https://mlsanity.com/paper/2310.04468/) (2023)
- [Beyond Accuracy: Automated De-Identification of Large Real-World Clinical Text Datasets](https://mlsanity.com/paper/2312.08495/) (2023)
- [Improving the Performance of Radiology Report De-identification with Large-Scale Training and Benchmarking Against Cloud Vendor Methods](https://mlsanity.com/paper/2511.04079/) (2025)
- [Jensen-Shannon Divergence Message-Passing for Rich-Text Graph Representation Learning](https://mlsanity.com/paper/2512.20094/) (2025)

## Built on by (most cited first, 1 of 1)

- [MedDeID enables locally governed clinical-text de-identification from real or synthetic training data](https://mlsanity.com/paper/2609.10049/) (2026)

## Abstract (arXiv)

De-identification of clinical text is a prerequisite for the secondary use of electronic health records. Existing public benchmarks such as the i2b2 2006 and 2014 corpora are over a decade old and lack the semantic and demographic diversity of modern clinical narratives. Large Language Models (LLMs) reach state-of-the-art zero-shot extraction, but their use at enterprise scale is limited by computational cost and by hospital data governance that restricts sending Protected Health Information (PHI) to cloud APIs. We introduce SHIELD (Synthetic Human-annotated Identifier-replaced Entries for Learning and De-identification), a diverse clinical note dataset of 1,381 notes with 10,229 gold-standard PHI spans across 9 categories, built with set-cover diversity sampling across demographic and document-type strata and human-in-the-loop adjudication. We evaluate four LLMs (two proprietary, two open-weight) to establish a performance ceiling on SHIELD, then show that a teacher-student distillation framework transfers these capabilities into locally deployable Small Language Models. Our best distilled model reaches micro-averaged span-level precision of 0.89 and recall of 0.88 while running on standard workstation hardware. It trails its cloud teacher on per-category recall (0.90 vs. 0.81 macro-averaged) but remains competitive given its lower cost and on-premise deployability. Cross-dataset evaluation shows that diversity-trained models generalize well on universal structured PHI categories, while institution-specific entities remain hard to transfer in both directions, which suggests pairing broad-coverage models with specialized models for high-volume, semi-structured note types. We publicly release the SHIELD dataset and the distilled DeBERTa v3 model to provide an accurate, cost-effective de-identification pipeline deployable entirely behind institutional firewalls.