mlsanityData to 14 Oct 2026

How to Train Private Clinical Language Models: A Comparative Study of Privacy-Preserving Pipelines for ICD-9 Coding

arXiv 2511.14936 ↗First posted 18 Nov 2025cs.LG cs.CLcited by 0

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Large language models trained on clinical text risk exposing sensitive patient information, yet differential privacy (DP) methods often severely degrade the diagnostic accuracy needed for deployment. Despite rapid progress in DP optimisation and text generation, it remains unclear which privacy-preserving strategy actually works best for clinical language tasks. We present the first systematic head-to-head comparison of four training pipelines for automated diagnostic coding from hospital discharge summaries. All pipelines use identical 1B-parameter models and matched privacy budgets to predict ICD-9 codes. At moderate and relaxed privacy budgets (varepsilon ∈ 4, 6), knowledge distillation from DP-trained teachers outperforms both direct DP-SGD and DP-synthetic data training, recovering up to 63% of the non-private performance whilst maintaining strong empirical privacy (membership-inference AUC ≈ 0.5). These findings expose large differences in the privacy-utility trade-off across architectures and identify knowledge distillation as the most practical route to privacy-preserving clinical NLP.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 13 of its 15 references that are in the map. First posted on arXiv 18 Nov 2025.

Builds on

13 of 15 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Differentially Private Knowledge Distillation via Synthetic Text Generation1 Mar

Other references · 12

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2016Deep Learning with Differential Privacycited by 2,728
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  4. 2020Extracting Training Data from Large Language Modelscited by 2,189
  5. 2021Membership Inference Attacks From First Principlescited by 878
  6. 2018Born Again Neural Networkscited by 546
  7. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  8. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 383
  9. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  10. 2023On student-teacher deviations in distillation: does it pay to disobey?cited by 16
  11. 2024LoRA Provides Differential Privacy by Design via Random Sketchingcited by 3
  12. 2024Can large language models be privacy preserving and fair medical coders?cited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Training a Large Language Model for Medical Coding Using Privacy-Preserving Synthetic Clinical Data
  2. 2025From Learning to Unlearning: Biomedical Security Protection in Multimodal Large Language Models
  3. 2026MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Medical Open-Ended Question Answering
  4. 2025Leveraging Open-Source Large Language Models for Clinical Information Extraction in Resource-Constrained Settings
  5. 2025Exploring Membership Inference Vulnerabilities in Clinical Large Language Models
  6. 2026Fidelity, Diversity, and Privacy: A Multi-Dimensional LLM Evaluation for Clinical Data Augmentation
  7. 2026Leveraging Large Language Models to Extract and Translate Medical Information in Doctors' Notes for Health Records and Diagnostic Billing Codes
  8. 2024PrecLLM: A Privacy-Preserving Framework for Efficient Clinical Annotation Extraction from Unstructured EHRs using Small-Scale LLMs