mlsanityData to 14 Oct 2026

Pop Quiz Attack: Black-box Membership Inference Attacks Against Large Language Models

arXiv 2605.06423 ↗First posted 7 May 2026cs.CRcited by 0

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Large language models (LLMs) show strong performance across many applications, but their ability to memorize and potentially reveal training data raises serious privacy concerns. We introduce the PopQuiz Attack, a black-box membership inference attack that tests whether a model can recall specific training examples. The core idea is to turn target data into quiz-style multiple-choice questions and infer membership from the model's answers. Across six widely used LLMs (GPT-3.5, GPT-4o, LLaMA2-7b, LLaMA2-13b, Mistral-7b, and Vicuna-7b) and four datasets, our method achieves an average ROC-AUC of 0.873 and outperforms existing approaches by 20.6%. We further analyze factors affecting attack success, including query complexity, data type, data structure, and training settings. We also evaluate instruction-based, filter-based, and differential privacy-based defenses, which reduce performance but do not eliminate the risk. Our results highlight persistent privacy vulnerabilities in modern LLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 36 of its 47 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

36 of 47 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Membership Inference Attacks Against In-Context Learning2 Sep
  2. 2024Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment8 Jul
  3. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods23 Jun
  4. 2024Do Membership Inference Attacks Work on Large Language Models?12 Feb
  5. 2023Did the Neurons Read your Book? Document-level Membership Inference for Large Language Models23 Oct
  6. 2023Analyzing Leakage of Personally Identifiable Information in Language Models1 Feb
  7. 2021Membership Inference Attacks From First Principles7 Dec
  8. 2021Enhanced Membership Inference Attacks against Machine Learning Models18 Nov
  9. 2021Membership Inference Attacks on Machine Learning: A Survey14 Mar
  10. 2021ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models4 Feb

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  4. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  5. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  6. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  7. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 498
  8. 2020Label-Only Membership Inference Attackscited by 388
  9. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 268
  10. 2020Information Leakage in Embedding Modelscited by 203
  11. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  12. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 141
14 more
  1. 2023On the Risk of Misinformation Pollution with Large Language Modelscited by 128
  2. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
  3. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
  4. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 93
  5. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 93
  6. 2021Node-Level Membership Inference Attacks Against Graph Neural Networkscited by 84
  7. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 83
  8. 2023Composite Backdoor Attacks Against Large Language Modelscited by 73
  9. 2024Datasets for Large Language Models: A Comprehensive Surveycited by 71
  10. 2023In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPTcited by 37
  11. 2022Amplifying Membership Exposure via Data Poisoningcited by 34
  12. 2023Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibilitycited by 30
  13. 2024MIA-Tuner: Adapting Large Language Models as Pre-training Text Detectorcited by 15
  14. 2023Which is better? Exploring Prompting Strategy For LLM-based Metricscited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
  2. 2026Detecting Data Contamination in Large Language Models
  3. 2024Order of Magnitude Speedups for LLM Membership Inference
  4. 2026On the Privacy of LLMs: An Ablation Study
  5. 2024Pandora's White-Box: Precise Training Data Detection and Extraction in Large Language Models
  6. 2024On the Privacy Risk of In-context Learning
  7. 2025Membership Inference Attacks on Large-Scale Models: A Survey
  8. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration