mlsanityData to 7 Oct 2026

Towards the Anonymization of the Language Modeling

arXiv 2501.02407 ↗First posted 5 Jan 2025cs.CL cs.CR cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Rapid advances in Natural Language Processing (NLP) have revolutionized many fields, including healthcare. However, these advances raise significant privacy concerns, especially when pre-trained models fine-tuned and specialized on sensitive data can memorize and then expose and regurgitate personal information. This paper presents a privacy-preserving language modeling approach to address the problem of language models anonymization, and thus promote their sharing. Specifically, we propose both a Masking Language Modeling (MLM) methodology to specialize a BERT-like language model, and a Causal Language Modeling (CLM) methodology to specialize a GPT-like model that avoids the model from memorizing direct and indirect identifying information present in the training data. We have comprehensively evaluated our approaches using a medical dataset and compared them against different baselines. Our results indicate that by avoiding memorizing both direct and indirect identifiers during model specialization, our masking and causal language modeling schemes offer a good tradeoff for maintaining high privacy while retaining high utility.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 67 references that are in the map. First posted on arXiv 5 Jan 2025.

Builds on

30 of 67 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2016Deep Learning with Differential Privacycited by 2,682
  5. 2020Extracting Training Data from Large Language Modelscited by 2,144
  6. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  7. 2021Membership Inference Attacks From First Principlescited by 839
  8. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  9. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  10. 2019How to Fine-Tune BERT for Text Classification?cited by 409
  11. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  12. 2019CamemBERT: a Tasty French Language Modelcited by 368
18 more
  1. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
  2. 2021Enhanced Membership Inference Attacks against Machine Learning Modelscited by 237
  3. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  4. 2022The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymizationcited by 94
  5. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 91
  6. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 91
  7. 2021Membership Inference Attack Susceptibility of Clinical Language Modelscited by 50
  8. 2022RelaxLoss: Defending Membership Inference Attacks without Losing Utilitycited by 35
  9. 2024Evaluations of Machine Learning Privacy Defenses are Misleadingcited by 35
  10. 2023DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domainscited by 27
  11. 2024Learnable Privacy Neurons Localization in Language Modelscited by 26
  12. 2024The VoicePrivacy 2022 Challenge: Progress and Perspectives in Voice Anonymisationcited by 17
  13. 2021The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architecturescited by 16
  14. 2024Fine-Tuning Language Models with Differential Privacy through Adaptive Noise Allocationcited by 14
  15. 2023An Empirical Study on Using Large Language Models to Analyze Software Supply Chain Security Failurescited by 11
  16. 2022De-Identification of French Unstructured Clinical Notes for Machine Learning Taskscited by 3
  17. 2023Development and validation of a natural language processing algorithm to pseudonymize documents in the context of a clinical data warehousecited by 3
  18. 2023A Novel Approach to Identify Security Controls in Source Codecited by 1

Built on by

Not cited yet.