mlsanityData to 7 Oct 2026

EL-MIA: Quantifying Membership Inference Risks of Sensitive Entities in LLMs

arXiv 2511.00192 ↗First posted 31 Oct 2025cs.LG cs.AIcited by 0

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Membership inference attacks (MIA) aim to infer whether a particular data point is part of the training dataset of a model. In this paper, we propose a new task in the context of LLM privacy: entity-level discovery of membership risk focused on sensitive information (PII, credit card numbers, etc). Existing methods for MIA can detect the presence of entire prompts or documents in the LLM training data, but they fail to capture risks at a finer granularity. We propose the ``EL-MIA'' framework for auditing entity-level membership risks in LLMs. We construct a benchmark dataset for the evaluation of MIA methods on this task. Using this benchmark, we conduct a systematic comparison of existing MIA techniques as well as two newly proposed methods. We provide a comprehensive analysis of the results, trying to explain the relation of the entity level MIA susceptability with the model scale, training epochs, and other surface level factors. Our findings reveal that existing MIA methods are limited when it comes to entity-level membership inference of the sensitive attributes, while this susceptibility can be outlined with relatively straightforward methods, highlighting the need for stronger adversaries to stress test the provided threat model.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 37 of its 47 references that are in the map. First posted on arXiv 31 Oct 2025.

Builds on

37 of 47 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Privacy Auditing of Large Language Models9 Mar
  2. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)25 Jun
  3. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods23 Jun
  4. 2024Semantic Membership Inference Attack against Large Language Models14 Jun
  5. 2024LLM Dataset Inference: Did you train on my dataset?10 Jun
  6. 2024Sampling-based Pseudo-Likelihood for Membership Inference Attacks17 Apr
  7. 2024Do Membership Inference Attacks Work on Large Language Models?12 Feb
  8. 2023Did the Neurons Read your Book? Document-level Membership Inference for Large Language Models23 Oct
  9. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparison29 May
  10. 2023Analyzing Leakage of Personally Identifiable Information in Language Models1 Feb

Other references · 27

  1. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  2. 2020Extracting Training Data from Large Language Modelscited by 2,144
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  4. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  6. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  7. 2023Large Language Modelscited by 774
  8. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  9. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  10. 2023ProPILE: Probing Privacy Leakage in Large Language Modelscited by 127
  11. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  12. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 123
15 more
  1. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 102
  2. 2024On Protecting the Data Privacy of Large Language Models (LLMs): A Surveycited by 91
  3. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 72
  4. 2023Exploring Memorization in Fine-tuned Language Modelscited by 44
  5. 2024Measuring memorization in language models via probabilistic extractioncited by 40
  6. 2024Proving membership in LLM pretraining data via data watermarkscited by 38
  7. 2024Copyright Traps for Large Language Modelscited by 36
  8. 2024Probing Language Models for Pre-training Data Detectioncited by 24
  9. 2024Data Contamination Calibration for Black-box LLMscited by 23
  10. 2022Canary Extraction in Natural Language Understanding Modelscited by 20
  11. 2024Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decodingcited by 20
  12. 2024Recent Advances in Named Entity Recognition: A Comprehensive Survey and Comparative Studycited by 19
  13. 2023Quantifying and Analyzing Entity-level Memorization in Large Language Modelscited by 18
  14. 2024Adaptive Pre-training Data Detection for Large Language Models via Surprising Tokenscited by 10
  15. 2025Can Large Language Models Really Recognize Your Name?cited by 4

Built on by

Not cited yet.