mlsanityData to 13 Oct 2026

Analyzing Information Leakage of Updates to Natural Language Models

arXiv 1912.07942 ↗First posted 17 Dec 2019cs.LG cs.CL cs.CRcited by 90

Not on the map. Why?

Cited by 31 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

To continuously improve quality and reflect changes in data, machine learning applications have to regularly retrain and update their core models. We show that a differential analysis of language model snapshots before and after an update can reveal a surprising amount of detailed information about changes in the training data. We propose two new metrics---differential score and differential rank---for analyzing the leakage due to updates of natural language models. We perform leakage analysis using these metrics across models trained on several different datasets using different methods and configurations. We discuss the privacy implications of our findings, propose mitigation strategies and evaluate their effect.

Key facts. Cited by 90 papers in the map (data through 13 Oct 2026). First posted on arXiv 17 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 11; 2022: 29; 2023: 21; 2024: 18; 2025: 8; 2026: 3.

90 papers.

Sort:

By area:

  1. 2023Detecting Pretraining Data from Large Language Modelscited by 383
  2. 2022A Survey of Machine Unlearningcited by 311
  3. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 284
  4. 2021Unrolling SGD: Understanding Factors Influencing Machine Unlearningcited by 277
  5. 2021Machine Unlearning of Features and Labelscited by 229
  6. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  7. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 195
  8. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 185
  9. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 173
  10. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  11. 2022Reconstructing Training Data with Informed Adversariescited by 147
  12. 2022Measuring Forgetting of Memorized Training Examplescited by 118
38 more
  1. 2022Truth Serum: Poisoning Machine Learning Models to Reveal Their Secretscited by 101
  2. 2021Hidden Backdoors in Human-Centric Language Modelscited by 90
  3. 2022Recommendation Unlearningcited by 80
  4. 2022Do Language Models Plagiarize?cited by 73
  5. 2022Recovering Private Text in Federated Learning of Language Modelscited by 70
  6. 2023Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermarkcited by 67
  7. 2023DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Passcited by 66
  8. 2024Building Guardrails for Large Language Modelscited by 64
  9. 2022"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practicecited by 56
  10. 2022Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Modelscited by 53
  11. 2022SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learningcited by 51
  12. 2024Evaluating Language Model Agency through Negotiationscited by 46
  13. 2022Memorization in NLP Fine-tuning Methodscited by 45
  14. 2022Hidden Poison: Machine Unlearning Enables Camouflaged Poisoning Attackscited by 43
  15. 2022Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queriescited by 40
  16. 2021Quantifying and Mitigating Privacy Risks of Contrastive Learningcited by 36
  17. 2023Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimationcited by 36
  18. 2021Membership Inference on Word Embedding and Beyondcited by 35
  19. 2024Machine Unlearning: A Comprehensive Surveycited by 35
  20. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
  21. 2022Bayesian Estimation of Differential Privacycited by 32
  22. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  23. 2024Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defensescited by 30
  24. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  25. 2025Protecting Users From Themselves: Safeguarding Contextual Privacy in Interactions with Conversational Agentscited by 26
  26. 2022Deletion Inference, Reconstruction, and Compliance in Machine (Un)Learningcited by 24
  27. 2024Data Contamination Calibration for Black-box LLMscited by 24
  28. 2023Machine Unlearning: A Surveycited by 19
  29. 2021A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communicationcited by 18
  30. 2023Quantifying and Analyzing Entity-level Memorization in Large Language Modelscited by 18
  31. 2025The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Textcited by 18
  32. 2022Membership Inference Attacks and Generalization: A Causal Perspectivecited by 17
  33. 2023BDMMT: Backdoor Sample Detection for Language Models through Model Mutation Testingcited by 16
  34. 2025Differentially Private Steering for Large Language Model Alignmentcited by 15
  35. 2023The Inadequacy of Similarity-based Privacy Metrics: Privacy Attacks against "Truly Anonymous" Synthetic Datasetscited by 14
  36. 2024Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directionscited by 14
  37. 2024Measuring memorization in RLHF for code completioncited by 13
  38. 2022You Don't Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers' Private Personascited by 11

50 most cited of 90 All 90 (JSON)