mlsanityData to 7 Oct 2026

Membership Inference Attack Susceptibility of Clinical Language Models

arXiv 2104.08305 ↗First posted 16 Apr 2021cs.CLcited by 50

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Deep Neural Network (DNN) models have been shown to have high empirical privacy leakages. Clinical language models (CLMs) trained on clinical data have been used to improve performance in biomedical natural language processing tasks. In this work, we investigate the risks of training-data leakage through white-box or black-box access to CLMs. We design and employ membership inference attacks to estimate the empirical privacy leaks for model architectures like BERT and GPT2. We show that membership inference attacks on CLMs lead to non-trivial privacy leakages of up to 7%. Our results show that smaller models have lower empirical privacy leakages than larger ones, and masked LMs have lower leakages than auto-regressive LMs. We further show that differentially private CLMs can have improved model utility on clinical domain while ensuring low empirical privacy leakage. Lastly, we also study the effects of group-level membership inference and disease rarity on CLM privacy leakages.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 15 of its 25 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

15 of 25 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2016Deep Learning with Differential Privacycited by 2,682
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  10. 2019Publicly Available Clinical BERT Embeddingscited by 995
  11. 2017Renyi Differential Privacycited by 643
  12. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
3 more
  1. 2018Scalable Private Learning with PATEcited by 266
  2. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 149
  3. 2019Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?cited by 68

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 12; 2024: 16; 2025: 16; 2026: 3.

50 papers.

Sort:

By area:

  1. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  2. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  3. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  4. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
  5. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  6. 2023Privacy Issues in Large Language Models: A Surveycited by 57
  7. 2024Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Methodcited by 54
  8. 2023Did the Neurons Read your Book? Document-level Membership Inference for Large Language Modelscited by 51
  9. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  10. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  11. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 38
  12. 2022Differentially Private Decoding in Large Language Modelscited by 34
38 more
  1. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  2. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  3. 2023Integration of Large Language Models and Federated Learningcited by 28
  4. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  5. 2023MoPe: Model Perturbation-based Privacy Attacks on Language Modelscited by 25
  6. 2024Semantic Membership Inference Attack against Large Language Modelscited by 17
  7. 2025Membership Inference Attacks on Large-Scale Models: A Surveycited by 17
  8. 2024De-identification is not enough: a comparison between de-identified and synthetic clinical notescited by 14
  9. 2024Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directionscited by 14
  10. 2024Detecting Training Data of Large Language Models via Expectation Maximizationcited by 13
  11. 2025Membership Inference Attacks as Privacy Tools: Reliability, Disparity and Ensemblecited by 11
  12. 2024SoK: Reducing the Vulnerability of Fine-tuned Language Models to Membership Inference Attackscited by 10
  13. 2023Gradient-Free Privacy Leakage in Federated Language Models through Selective Weight Tamperingcited by 9
  14. 2025SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attackscited by 9
  15. 2023Assessing Privacy Risks in Language Models: A Case Study on Summarization Taskscited by 8
  16. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  17. 2023Improved Membership Inference Attacks Against Language Classification Modelscited by 7
  18. 2025Synthetic Data Privacy Metricscited by 7
  19. 2025LoRA-Leak: Membership Inference Attacks Against LoRA Fine-tuned Language Modelscited by 7
  20. 2024Forget to Flourish: Leveraging Machine-Unlearning on Pretrained Language Models for Privacy Leakagecited by 6
  21. 2025Can Differentially Private Fine-tuning LLMs Protect Against Privacy Attacks?cited by 4
  22. 2025Quantifying the Privacy Implications of High-Fidelity Synthetic Network Trafficcited by 3
  23. 2026Do LLMs Really Memorize Personally Identifiable Information? Revisiting PII Leakage with a Cue-Controlled Memorization Frameworkcited by 3
  24. 2024Quantifying Memorization and Detecting Training Data of Pre-trained Language Models using Japanese Newspapercited by 2
  25. 2024SoK: Security and Privacy Risks of Healthcare AIcited by 2
  26. 2022Swing Distillation: A Privacy-Preserving Knowledge Distillation Frameworkcited by 1
  27. 2025On Membership Inference Attacks in Knowledge Distillationcited by 1
  28. 2025Data Compressibility Quantifies LLM Memorizationcited by 1
  29. 2025Exploring Membership Inference Vulnerabilities in Clinical Large Language Modelscited by 1
  30. 2025Leverage Unlearning to Sanitize LLMscited by 1
  31. 2024Inside the Black Box: Detecting Data Leakage in Pre-trained Language Encoderscited by 0
  32. 2025Towards the Anonymization of the Language Modelingcited by 0
  33. 2025Language modelling techniques for analysing the impact of human genetic variationcited by 0
  34. 2025Fragments to Facts: Partial-Information Fragment Inference from LLMscited by 0
  35. 2025The Model's Language Matters: A Comparative Privacy Analysis of LLMscited by 0
  36. 2025Membership Inference Attack against Large Language Model-based Recommendation Systems: A New Distillation-based Paradigmcited by 0
  37. 2026MC-PDD: Masked Corpus-Level Pretraining Data Detection for Black-Box Large Language Modelscited by 0
  38. 2026Clinically Grounded Privacy Evaluation of Medical LMscited by 0