# Unintended Memorization and Timing Attacks in Named Entity Recognition Models

arXiv: https://arxiv.org/abs/2211.02245 · HTML: https://mlsanity.com/paper/2211.02245/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Membership inference attacks, one of 12 directions in LLM safety and security.
- Builds on 28 of its 74 references that are in the map.
- First posted on arXiv 4 Nov 2022.

## Placed in

- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [Label-Only Membership Inference Attacks](https://mlsanity.com/paper/2007.14321/) (2020)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [BAE: BERT-based Adversarial Examples for Text Classification](https://mlsanity.com/paper/2004.01970/) (2020)
- [Information Leakage in Embedding Models](https://mlsanity.com/paper/2004.00053/) (2020)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning](https://mlsanity.com/paper/1904.01067/) (2019)
- [A Survey on Recent Advances in Named Entity Recognition from Deep Learning models](https://mlsanity.com/paper/1910.11470/) (2019)
- [Deep Text Classification Can be Fooled](https://mlsanity.com/paper/1704.08006/) (2017)
- [Analyzing Information Leakage of Updates to Natural Language Models](https://mlsanity.com/paper/1912.07942/) (2019)
- [De-identification of Patient Notes with Recurrent Neural Networks](https://mlsanity.com/paper/1606.03475/) (2016)
- [Stealing Neural Networks via Timing Side Channels](https://mlsanity.com/paper/1812.11720/) (2018)
- [On the (In)Feasibility of Attribute Inference Attacks on Machine Learning Models](https://mlsanity.com/paper/2103.07101/) (2021)
- [FedNER: Privacy-preserving Medical Named Entity Recognition with Federated Learning](https://mlsanity.com/paper/2003.09288/) (2020)
- [Efficient Adaptation of Pretrained Transformers for Abstractive Summarization](https://mlsanity.com/paper/1906.00138/) (2019)
- [AMALGUM -- A Free, Balanced, Multilayer English Web Corpus](https://mlsanity.com/paper/2006.10677/) (2020)
- [Few-Shot Sequence Labeling with Label Dependency Transfer and Pair-wise Embedding](https://mlsanity.com/paper/1906.08711/) (2019)
- [An Analysis Of Protected Health Information Leakage In Deep-Learning Based De-Identification Algorithms](https://mlsanity.com/paper/2101.12099/) (2021)

## Built on by (most cited first, 2 of 2)

- [Activation Functions Considered Harmful: Recovering Neural Network Weights through Controlled Channels](https://mlsanity.com/paper/2503.19142/) (2025)
- [Exploring prompts to elicit memorization in masked language model-based named entity recognition](https://mlsanity.com/paper/2405.03004/) (2024)

## Abstract (arXiv)

Named entity recognition models (NER), are widely used for identifying named entities (e.g., individuals, locations, and other information) in text documents. Machine learning based NER models are increasingly being applied in privacy-sensitive applications that need automatic and scalable identification of sensitive information to redact text for data sharing. In this paper, we study the setting when NER models are available as a black-box service for identifying sensitive information in user documents and show that these models are vulnerable to membership inference on their training datasets. With updated pre-trained NER models from spaCy, we demonstrate two distinct membership attacks on these models. Our first attack capitalizes on unintended memorization in the NER's underlying neural network, a phenomenon NNs are known to be vulnerable to. Our second attack leverages a timing side-channel to target NER models that maintain vocabularies constructed from the training data. We show that different functional paths of words within the training dataset in contrast to words not previously seen have measurable differences in execution time. Revealing membership status of training samples has clear privacy implications, e.g., in text redaction, sensitive words or phrases to be found and removed, are at risk of being detected in the training dataset. Our experimental evaluation includes the redaction of both password and health data, presenting both security risks and privacy/regulatory issues. This is exacerbated by results that show memorization with only a single phrase. We achieved 70% AUC in our first attack on a text redaction use-case. We also show overwhelming success in the timing attack with 99.23% AUC. Finally we discuss potential mitigation approaches to realize the safe use of NER models in light of the privacy and security implications of membership inference attacks.