mlsanityData to 14 Oct 2026

Privacy Auditing of Large Language Models

arXiv 2503.06808 ↗First posted 9 Mar 2025cs.CR cs.AI cs.LGcited by 23

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Current techniques for privacy auditing of large language models (LLMs) have limited efficacy -- they rely on basic approaches to generate canaries which leads to weak membership inference attacks that in turn give loose lower bounds on the empirical privacy leakage. We develop canaries that are far more effective than those used in prior work under threat models that cover a range of realistic settings. We demonstrate through extensive experiments on multiple families of fine-tuned LLMs that our approach sets a new standard for detection of privacy leakage. For measuring the memorization rate of non-privately trained LLMs, our designed canaries surpass prior approaches. For example, on the Qwen2.5-0.5B model, our designed canaries achieve 49.6% TPR at 1% FPR, vastly surpassing the prior approach's 4.2% TPR at 1% FPR. Our method can be used to provide a privacy audit of varepsilon ≈ 1 for a model trained with theoretical varepsilon of 4. To the best of our knowledge, this is the first time that a privacy audit of LLM training has achieved nontrivial auditing success in the setting where the attacker cannot train shadow models, insert gradient canaries, or access the model at every iteration.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 61 of its 80 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

61 of 80 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Nob-MIAs: Non-biased Membership Inference Attacks Assessment on Large Language Models with Ex-Post Dataset Construction12 Aug
  2. 2024Blind Baselines Beat Membership Inference Attacks for Foundation Models23 Jun
  3. 2024PANORAMIA: Privacy Auditing of Machine Learning Models without Retraining12 Feb
  4. 2024Do Membership Inference Attacks Work on Large Language Models?12 Feb
  5. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparison29 May
  6. 2023Students Parrot Their Teachers: Membership Inference on Model Distillation6 Mar
  7. 2023Analyzing Leakage of Personally Identifiable Information in Language Models1 Feb
  8. 2021Membership Inference Attacks From First Principles7 Dec

Other references · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2023Mistral 7Bcited by 3,164
  10. 2016Deep Learning with Differential Privacycited by 2,728
  11. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  12. 2020Extracting Training Data from Large Language Modelscited by 2,189
38 more
  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2023PaLM 2 Technical Reportcited by 1,162
  4. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  5. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  6. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  7. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  8. 2023Extracting Training Data from Diffusion Modelscited by 732
  9. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  10. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  11. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  12. 2020Label-Only Membership Inference Attackscited by 388
  13. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  14. 2013The Composition Theorem for Differential Privacycited by 297
  15. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  16. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  17. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 218
  18. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  19. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 199
  20. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 181
  21. 2024Stealing Part of a Production Language Modelcited by 159
  22. 2023Emergent and Predictable Memorization in Large Language Modelscited by 148
  23. 2022The Privacy Onion Effect: Memorization is Relativecited by 136
  24. 2023Privacy Auditing with One (1) Training Runcited by 123
  25. 2021Large-Scale Differentially Private BERTcited by 118
  26. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  27. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 100
  28. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
  29. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 83
  30. 2023Language Model Inversioncited by 68
  31. 2023DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineercited by 57
  32. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  33. 2023Privacy-Preserving In-Context Learning for Large Language Modelscited by 50
  34. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  35. 2022Debugging Differential Privacy: A Case Study for Privacy Auditingcited by 40
  36. 2023User Inference Attacks on Large Language Modelscited by 36
  37. 2024Teach LLMs to Phish: Stealing Private Information from Language Modelscited by 32
  38. 2024Nearly Tight Black-Box Auditing of Differentially Private Machine Learningcited by 21

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 7.

23 papers.

Sort:

By area:

  1. 2025Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agentscited by 11
  2. 2025Rectifying Privacy and Efficacy Measurements in Machine Unlearning: A New Inference Attack Perspectivecited by 9
  3. 2025CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfittingcited by 7
  4. 2025Optimizing Canaries for Privacy Auditing with Metagradient Descentcited by 6
  5. 2025Train Once, Answer All: Many Pretraining Experiments for the Cost of Onecited by 6
  6. 2025The Hitchhiker's Guide to Efficient, End-to-End, and Tight DP Auditingcited by 5
  7. 2025Can Differentially Private Fine-tuning LLMs Protect Against Privacy Attacks?cited by 4
  8. 2025What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requestscited by 4
  9. 2025ContextLeak: Auditing Leakage in Private In-Context Learning Methodscited by 4
  10. 2026Benchmarking Empirical Privacy Protection for Adaptations of Large Language Modelscited by 4
  11. 2025Empirical Privacy Variancecited by 3
  12. 2025A Unified Framework for Adversary-Aware Differential Privacy Boundscited by 2
11 more
  1. 2025Efficient Machine Unlearning by Model Splitting and Core Sample Selectioncited by 1
  2. 2026Protecting patient privacy in clinical foundation models: Technical and legal perspectivescited by 1
  3. 2026ToolMinimize: Auditing and Rewriting LLM Agent Tool Calls to Minimize Privacy Exposurecited by 1
  4. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0
  5. 2025f-INE: A Hypothesis Testing Framework for Estimating Influence under Training Randomnesscited by 0
  6. 2025EL-MIA: Quantifying Membership Inference Risks of Sensitive Entities in LLMscited by 0
  7. 2025Black-Box Auditing of Quantum Model: Lifted Differential Privacy with Quantum Canariescited by 0
  8. 2026White-Box Sensitivity Auditing with Steering Vectorscited by 0
  9. 2026From Fragmentation to Integration: Exploring the Design Space of AI Agents for Human-as-the-Unit Privacy Managementcited by 0
  10. 2026DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuningcited by 0
  11. 2026Natural Identifiers for Privacy and Data Audits in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Privacy Risks and Protection in Large Language Models
  2. 2026LLM-CEG: Extending the Classification Error Gauge Framework for Privacy Auditing of Large Language Models ↗
  3. 2026Advancing the State-of-the-Art in Empirical Privacy Auditing
  4. 2024Privacy-Preserving Large Language Models: Mechanisms, Applications, and Future Directions
  5. 2024Preserving Privacy in Large Language Models: A Survey on Current Threats and Solutions
  6. 2025The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text
  7. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
  8. 2024Order of Magnitude Speedups for LLM Membership Inference