mlsanityData to 13 Oct 2026

The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text

arXiv 2502.14921 ↗First posted 19 Feb 2025cs.CL cs.CR cs.LGcited by 18

On the map: Synthetic data and self-improvement for LLMs → Privacy-preserving tabular data synthesis

Abstract (arXiv)

How much information about training samples can be leaked through synthetic data generated by Large Language Models (LLMs)? Overlooking the subtleties of information flow in synthetic data generation pipelines can lead to a false sense of privacy. In this paper, we assume an adversary has access to some synthetic data generated by a LLM. We design membership inference attacks (MIAs) that target the training data used to fine-tune the LLM that is then used to synthesize data. The significant performance of our MIA shows that synthetic data leak information about the training data. Further, we find that canaries crafted for model-based MIAs are sub-optimal for privacy auditing when only synthetic data is released. Such out-of-distribution canaries have limited influence on the model's output when prompted to generate useful, in-distribution synthetic data, which drastically reduces their effectiveness. To tackle this problem, we leverage the mechanics of auto-regressive models to design canaries with an in-distribution prefix and a high-perplexity suffix that leave detectable traces in synthetic data. This enhances the power of data-based MIAs and provides a better assessment of the privacy risks of releasing synthetic data generated by LLMs.

Key facts. Cited by 18 papers in the map (data through 13 Oct 2026). Placed in Privacy-preserving tabular data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 47 of its 62 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

47 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Differentially Private Synthetic Data via Foundation Model APIs 2: Text4 Mar

Other references · 46

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
  3. 2016Deep Learning with Differential Privacycited by 2,718
  4. 2015Character-level Convolutional Networks for Text Classificationcited by 2,319
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,273
  6. 2020Extracting Training Data from Large Language Modelscited by 2,181
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,841
  8. 2021Membership Inference Attacks From First Principlescited by 872
  9. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  10. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  11. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 495
  12. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 427
34 more
  1. 2020Label-Only Membership Inference Attackscited by 386
  2. 2023Detecting Pretraining Data from Large Language Modelscited by 383
  3. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 378
  4. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  5. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 270
  6. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 255
  7. 2021Enhanced Membership Inference Attacks against Machine Learning Modelscited by 247
  8. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 218
  9. 2019Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inferencecited by 174
  10. 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 144
  11. 2022The Privacy Onion Effect: Memorization is Relativecited by 136
  12. 2019Reconstruction and Membership Inference Attacks against Generative Modelscited by 122
  13. 2020Synthetic Data -- Anonymisation Groundhog Daycited by 108
  14. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  15. 2023Low-Cost High-Power Membership Inference Attackscited by 106
  16. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
  17. 2019Analyzing Information Leakage of Updates to Natural Language Modelscited by 90
  18. 2022Differentially Private Language Models for Secure Data Sharingcited by 68
  19. 2023Harnessing large-language models to generate private synthetic textcited by 65
  20. 2022TAPAS: a Toolbox for Adversarial Privacy Auditing of Synthetic Datacited by 57
  21. 2023Did the Neurons Read your Book? Document-level Membership Inference for Large Language Modelscited by 56
  22. 2022SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learningcited by 51
  23. 2023Privacy-Preserving In-Context Learning for Large Language Modelscited by 50
  24. 2024Proving membership in LLM pretraining data via data watermarkscited by 40
  25. 2024Copyright Traps for Large Language Modelscited by 37
  26. 2022Defending against Reconstruction Attacks with R'enyi Differential Privacycited by 32
  27. 2022Bayesian Estimation of Differential Privacycited by 32
  28. 2023Achilles' Heels: Vulnerable Record Identification in Synthetic Data Publishingcited by 28
  29. 2023MoPe: Model Perturbation-based Privacy Attacks on Language Modelscited by 25
  30. 2024Private prediction for large-scale synthetic text generationcited by 24
  31. 2024Context-Aware Membership Inference Attacks against Pre-trained Large Language Modelscited by 22
  32. 2023Investigating the Effect of Misalignment on Membership Privacy in the White-box Settingcited by 9
  33. 2021Differentially Private n-gram Extractioncited by 8
  34. 2022An Empirical Study on the Membership Inference Attack against Tabular Data Synthesis Modelscited by 6

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 7; 2026: 11.

18 papers.

Sort:

By area:

  1. 2025InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacycited by 10
  2. 2025Optimizing Canaries for Privacy Auditing with Metagradient Descentcited by 6
  3. 2025Evaluating Differentially Private Generation of Domain-Specific Textcited by 5
  4. 2025Can We Infer Confidential Properties of Training Data from LLMs?cited by 4
  5. 2025SynBench: A Benchmark for Differentially Private Text Generationcited by 4
  6. 2026MIDST Challenge at SaTML 2025: Membership Inference over Diffusion-models-based Synthetic Tabular datacited by 2
  7. 2025Privacy-Preserving Generation of Clinical Narratives from Medical Terminologiescited by 1
  8. 2025Black-Box Auditing of Quantum Model: Lifted Differential Privacy with Quantum Canariescited by 0
  9. 2026MAPLE: Metadata Augmented Private Language Evolutioncited by 0
  10. 2026Spontaneous Persuasion: An Audit of Model Persuasiveness in Everyday Conversationscited by 0
  11. 2026Behavioral Canaries: Auditing Private Retrieved Context Usage in RL Fine-Tuningcited by 0
  12. 2026Advancing the State-of-the-Art in Empirical Privacy Auditingcited by 0
6 more
  1. 2026Phantoms and Disclosures: A Statistical Framework for Auditing Privacy in Synthetic Datacited by 0
  2. 2026Clinical Communication Processing with Models Trained on LLM-Generated Synthetic Data: A Structured Survey and Novel Application Case Studiescited by 0
  3. 2026Protecting patient privacy in clinical foundation models: Technical and legal perspectivescited by 0
  4. 2026Position: Privacy Is a Claim, Not a Property of Synthetic Datacited by 0
  5. 2026Subgroup Membership Inference Audits of Differentially Private Synthetic Textcited by 0
  6. 2026Canaries in the Bank: Auditing User-Level Privacy in Private Evolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Privacy Auditing Synthetic Data Release through Local Likelihood Attacks
  2. 2025Synth-MIA: A Testbed for Auditing Privacy Leakage in Tabular Data Synthesis
  3. 2025Privacy Auditing of Large Language Models
  4. 2024SafeSynthDP: Leveraging Large Language Models for Privacy-Preserving Synthetic Data Generation Using Differential Privacy
  5. 2025When Privacy Isn't Synthetic: Hidden Data Leakage in Generative AI Models
  6. 2025Ensembling Membership Inference Attacks Against Tabular Generative Models
  7. 2024Order of Magnitude Speedups for LLM Membership Inference
  8. 2023Synthetic is all you need: removing the auxiliary data assumption for membership inference attacks against synthetic data