mlsanityData to 14 Oct 2026

Have You Ever Seen Them? Entity-level Membership Inference through Interrogating Large Language Models

arXiv 2606.23030 ↗First posted 22 Jun 2026cs.CLcited by 0

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Large Language Models (LLMs) raise growing concerns about privacy leakage and copyright compliance. Membership inference is a key tool for assessing such risks, but existing studies mainly focus on whether specific samples or sample-based data units are used for training. We argue that LLMs exhibit a human-memory-like behavior: an LLM may not memorize a specific sample verbatim, yet it can accumulate and reveal knowledge about a real-world entity from scattered mentions. This analogy motivates us to examine whether an LLM can be interrogated like a human interviewee to reveal its exposure to entity-related information. Motivated by this question, we propose entity-level membership inference, which determines whether information related to a target entity is used in LLM training. We study this task in the practical label-only black-box setting, where only generated texts are observable. We formalize the task under clue, input, and model constraints, establish the necessary and sufficient conditions for its feasibility, and instantiate five interrogation strategies based on this formalization. The strategies use limited entity clues to construct prompts, elicit entity-related responses, and infer membership from semantic features among the generated texts. We construct entity-level datasets and adapt state-of-the-art sample-level label-only methods to the entity-level setting as baselines. Experiments on person entities show that our methods achieve AUC up to 0.97 and bring gains of 6.0%--17.5% in Balanced Accuracy over the best adapted baseline.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 37 of its 71 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

37 of 71 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025In-Context Probing for Membership Inference in Fine-Tuned Language Models18 Dec
  2. 2025Towards Label-Only Membership Inference Attack against Pre-trained Large Language Models26 Feb
  3. 2025Membership Inference Attacks Against Vision-Language Models27 Jan
  4. 2024Membership Inference Attacks Against In-Context Learning2 Sep
  5. 2024Is Difficulty Calibration All We Need? Towards More Practical Membership Inference Attacks31 Aug
  6. 2024LLM Dataset Inference: Did you train on my dataset?10 Jun
  7. 2024Do Membership Inference Attacks Work on Large Language Models?12 Feb
  8. 2023Did the Neurons Read your Book? Document-level Membership Inference for Large Language Models23 Oct
  9. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparison29 May
  10. 2022Membership Inference Attacks by Exploiting Loss Trajectory31 Aug
  11. 2021Membership Inference Attacks From First Principles7 Dec
  12. 2021Enhanced Membership Inference Attacks against Machine Learning Models18 Nov
1 more
  1. 2021On the Importance of Difficulty Calibration in Membership Inference Attacks15 Nov

Other references · 24

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2016Deep Learning with Differential Privacycited by 2,728
  8. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  12. 2016Neural Architectures for Named Entity Recognitioncited by 550
12 more
  1. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 498
  2. 2020Label-Only Membership Inference Attackscited by 388
  3. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  4. 2018A Survey on Deep Learning for Named Entity Recognitioncited by 282
  5. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 271
  6. 2022A Contrastive Framework for Neural Text Generationcited by 225
  7. 2019MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examplescited by 197
  8. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  9. 2019Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inferencecited by 175
  10. 2022Dataset Inference for Self-Supervised Modelscited by 36
  11. 2023User Inference Attacks on Large Language Modelscited by 36
  12. 2023Assessing Privacy Risks in Language Models: A Case Study on Summarization Taskscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Detecting Data Contamination in Large Language Models
  2. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods
  3. 2024Understanding Privacy Risks of Embeddings Induced by Large Language Models
  4. 2025What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests
  5. 2023Beyond Memorization: Violating Privacy Via Inference with Large Language Models
  6. 2025(Token-Level) InfoRMIA: Stronger Membership Inference and Memorization Assessment for LLMs
  7. 2026Detecting Non-Membership in LLM Training Data via Rank Correlations
  8. 2024Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decoding