mlsanityData to 14 Oct 2026

Canary Extraction in Natural Language Understanding Models

arXiv 2203.13920 ↗First posted 25 Mar 2022cs.CL cs.AI cs.IRcited by 21

Not on the map. Why?

Abstract (arXiv)

Natural Language Understanding (NLU) models can be trained on sensitive information such as phone numbers, zip-codes etc. Recent literature has focused on Model Inversion Attacks (ModIvA) that can extract training data from model parameters. In this work, we present a version of such an attack by extracting canaries inserted in NLU training data. In the attack, an adversary with open-box access to the model reconstructs the canaries contained in the model's training set. We evaluate our approach by performing text completion on canaries and demonstrate that by using the prefix (non-sensitive) tokens of the canary, we can generate the full canary. As an example, our attack is able to reconstruct a four digit code in the training dataset of the NLU model with a probability of 0.5 in its best configuration. As countermeasures, we identify several defense mechanisms that, when combined, effectively eliminate the risk of ModIvA in our experiments.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 14 of its 30 references that are in the map. First posted on arXiv 25 Mar 2022.

Builds on

14 of 30 references.

References · 14

  1. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  2. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  3. 2020Extracting Training Data from Large Language Modelscited by 2,189
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  5. 2017A Closer Look at Memorization in Deep Networkscited by 942
  6. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  7. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 377
  8. 2016End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRFcited by 341
  9. 2017Advances in Pre-Training Distributed Word Representationscited by 207
  10. 2020Information Leakage in Embedding Modelscited by 203
  11. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  12. 2019Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbationscited by 119
2 more
  1. 2017LOGAN: Membership Inference Attacks Against Generative Modelscited by 28
  2. 2019Membership Model Inversion Attacks for Deep Networkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 6; 2025: 6; 2026: 3.

21 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  2. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 285
  3. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 51
  4. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 45
  5. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 37
  6. 2024Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defensescited by 30
  7. 2024Text Embedding Inversion Security for Multilingual Language Modelscited by 26
  8. 2022CANIFE: Crafting Canaries for Empirical Privacy Measurement in Federated Learningcited by 24
  9. 2025A Survey on Model Extraction Attacks and Defenses for Large Language Modelscited by 19
  10. 2024The Mosaic Memory of Large Language Modelscited by 10
  11. 2024Privacy-preserving Fine-tuning of Large Language Models through Flatnesscited by 6
  12. 2025Rethinking the Privacy of Text Embeddings: A Reproducibility Study of "Text Embeddings Reveal (Almost) As Much As Text"cited by 6
9 more
  1. 2023Balancing Transparency and Risk: The Security and Privacy Risks of Open-Source Machine Learning Modelscited by 5
  2. 2026Unveiling Privacy Risks in Multi-modal Large Language Models: Task-specific Vulnerabilities and Mitigation Challengescited by 4
  3. 2024Defining 'Good': Evaluation Framework for Synthetic Smart Meter Datacited by 2
  4. 2025NLP Security and Ethics, in the Wildcited by 2
  5. 2025Beyond PII: How Users Attempt to Estimate and Mitigate Implicit LLM Inferencecited by 2
  6. 2025Auditing Information Disclosure During Large-Scale Gradient-Based Training via Gradient Uniquenesscited by 0
  7. 2025EL-MIA: Quantifying Membership Inference Risks of Sensitive Entities in LLMscited by 0
  8. 2026Responsible AI for General-Purpose Systems: Overview, Challenges, and A Path Forwardcited by 0
  9. 2026Beyond Homophily: Towards Generalized Graph Reconstruction Attack and Defensecited by 0