mlsanityData to 7 Oct 2026

An Investigation of Memorization Risk in Healthcare Foundation Models

arXiv 2510.12950 ↗First posted 14 Oct 2025cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Foundation models trained on large-scale de-identified electronic health records (EHRs) hold promise for clinical applications. However, their capacity to memorize patient information raises important privacy concerns. In this work, we introduce a suite of black-box evaluation tests to assess privacy-related memorization risks in foundation models trained on structured EHR data. Our framework includes methods for probing memorization at both the embedding and generative levels, and aims to distinguish between model generalization and harmful memorization in clinically relevant settings. We contextualize memorization in terms of its potential to compromise patient privacy, particularly for vulnerable subgroups. We validate our approach on a publicly available EHR foundation model and release an open-source toolkit to facilitate reproducible and collaborative privacy assessments in healthcare AI.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 14 Oct 2025.

Builds on

35 of 57 references.

References · 35

  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  3. 2020Extracting Training Data from Large Language Modelscited by 2,144
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  5. 2022Red Teaming Language Models with Language Modelscited by 983
  6. 2017A Closer Look at Memorization in Deep Networkscited by 936
  7. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  8. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  9. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  10. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  11. 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 509
  12. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
23 more
  1. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  2. 2020Label-Only Membership Inference Attackscited by 371
  3. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 368
  4. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  5. 2019BEHRT: Transformer for Electronic Health Recordscited by 258
  6. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  7. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  8. 2020Information Leakage in Embedding Modelscited by 199
  9. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  10. 2020A Survey of Privacy Attacks in Machine Learningcited by 143
  11. 2019Thieves on Sesame Street! Model Extraction of BERT-based APIscited by 136
  12. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 126
  13. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  14. 2023EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Modelscited by 101
  15. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 91
  16. 2019Inducing Relational Knowledge from BERTcited by 75
  17. 2024Zero Shot Health Trajectory Prediction Using Transformercited by 74
  18. 2023Language Model Inversioncited by 66
  19. 2020Differentially Private Language Models Benefit from Public Pre-trainingcited by 46
  20. 2023Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Eventscited by 46
  21. 2024Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Modelscited by 34
  22. 2024EHRMamba: Towards Generalizable and Scalable Foundation Models for Electronic Health Recordscited by 31
  23. 2023A Multi-Center Study on the Adaptability of a Shared Foundation Model for Electronic Health Recordscited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026EchoJEPA: A Latent Predictive Foundation Model for Echocardiographycited by 9
  2. 2026Pretrained, Frozen, Still Leaking: Auditing Cross-Encoder Attribute Transfer in EEG Foundation Modelscited by 1
  3. 2026Clinically Grounded Privacy Evaluation of Medical LMscited by 0
  4. 2026Protecting patient privacy in clinical foundation models: Technical and legal perspectivescited by 0
  5. 2026Memorisation bias in medical AIcited by 0