mlsanityData to 14 Oct 2026

Empirical Evaluation of Membership Inference Attacks on NLP Text Classifiers: A Baseline Study on SST-2

arXiv 2609.10935 ↗First posted 10 Sep 2026cs.CR cs.CL cs.LGcited by 0

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Membership inference attacks (MIAs) try to determine whether a specific record was used to train a model, a privacy risk that matters in natural language processing (NLP), where training data can contain sensitive user text. This paper presents a controlled benchmark of membership inference vulnerability for text classification on the GLUE SST-2 sentiment dataset. A TF-IDF + Logistic Regression pipeline and a fine-tuned DistilBERT classifier are compared under a loss-threshold MIA, with utility measured by development accuracy and macro F1. DistilBERT reached 0.9466 accuracy and 0.9460 macro F1 against 0.8756 and 0.8727 for Logistic Regression, yet both models leaked membership signal (Attack AUC 0.5615 and 0.5800, respectively). Two mitigations were tested. Stronger regularization reduced leakage for Logistic Regression at a visible utility cost, whereas fine-tuning DistilBERT for 2 epochs instead of 3 reduced leakage with negligible accuracy loss. Lightweight training adjustments can improve the privacy-utility trade-off without complex defenses.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 4 of its 10 references that are in the map. First posted on arXiv 10 Sep 2026.

Builds on

4 of 10 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Membership Inference Attacks on Large-Scale Models: A Survey25 Mar

Other references · 3

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Membership Inference Attacks as Privacy Tools: Reliability, Disparity and Ensemble
  2. 2026CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models
  3. 2025The Hidden Cost of Modeling P(X): Vulnerability to Membership Inference Attacks in Generative Text Classifiers
  4. 2024Scaling Up Membership Inference: When and How Attacks Succeed on Large Language Models
  5. 2026What Hard Tokens Reveal: Exploiting Low-confidence Tokens for Membership Inference Attacks against Large Language Models
  6. 2025Membership Inference Attacks on Tokenizers of Large Language Models
  7. 2024SoK: Reducing the Vulnerability of Fine-tuned Language Models to Membership Inference Attacks
  8. 2021Membership Inference Attacks on Machine Learning: A Survey