mlsanityData to 14 Oct 2026

Black-Box Membership Inference via Word-Level Probability Estimation

arXiv 2609.10611 ↗First posted 8 Sep 2026cs.CR cs.LG stat.MLcited by 0

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Membership inference attacks (MIAs) have emerged as critical tools for auditing privacy risks in large language models (LLMs), aiming to determine whether a given text was included in a model's training corpus. However, most existing MIAs require access to per-token logits or probabilities, making them inapplicable in practice to proprietary LLMs that expose only textual continuations. To address this underexplored setting, we propose Word-level Probability MIA (WPMIA), a statistically principled MIA for strict black-box privacy auditing. WPMIA estimates word-level generation probabilities via Monte Carlo sampling with local kernel smoothing, then aggregates these estimates into a sequence-level likelihood estimator. Furthermore, WPMIA constructs the likelihood conditioned on different prefixes, thereby amplifying the distributional differences between members and non-members. We evaluate WPMIA across various open-source LLMs and find that it consistently outperforms existing black-box baselines. Importantly, we also evaluate WPMIA on modern proprietary LLMs, including GPT-5-Chat, Gemini-2.5-Flash, and Claude-4.5-Haiku, achieving an average TPR@5%FPR of 42.0 across these models. These results offer a sound foundation for future research on strict black-box membership inference. Code is available at hrefhttps://github.com/niusj03/WPMIAhttps://github.com/niusj03/WPMIA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 21 of its 36 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

21 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Membership Inference on LLMs in the Wild16 Jan
  2. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods23 Jun
  3. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparison29 May

Other references · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  9. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  10. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  12. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
6 more
  1. 2023The Falcon Series of Open Language Modelscited by 535
  2. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 125
  3. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
  4. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 82
  5. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 74
  6. 2024Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decodingcited by 20

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Order of Magnitude Speedups for LLM Membership Inference
  2. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
  3. 2025Fast-MIA: Efficient and Scalable Membership Inference for LLMs
  4. 2025(Token-Level) InfoRMIA: Stronger Membership Inference and Memorization Assessment for LLMs
  5. 2026What Hard Tokens Reveal: Exploiting Low-confidence Tokens for Membership Inference Attacks against Large Language Models
  6. 2025Membership Inference Attacks on Large-Scale Models: A Survey
  7. 2025Privacy Auditing of Large Language Models
  8. 2026Powerful Training-Free Membership Inference Against Autoregressive Language Models