# Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration

arXiv: https://arxiv.org/abs/2311.06062 · HTML: https://mlsanity.com/paper/2311.06062/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- Placed in Membership inference attacks, one of 12 directions in LLM safety and security.
- Builds on 43 of its 73 references that are in the map.
- First posted on arXiv 10 Nov 2023.

## Placed in

- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 30 of 43)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [Label-Only Membership Inference Attacks](https://mlsanity.com/paper/2007.14321/) (2020)
- [Weight Poisoning Attacks on Pre-trained Models](https://mlsanity.com/paper/2004.06660/) (2020)
- [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [Membership Inference Attacks against Language Models via Neighbourhood Comparison](https://mlsanity.com/paper/2305.18462/) (2023)
- [Enhanced Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/2111.09679/) (2021)
- [Transformer Memory as a Differentiable Search Index](https://mlsanity.com/paper/2202.06991/) (2022)
- [Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://mlsanity.com/paper/2205.10770/) (2022)
- [MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examples](https://mlsanity.com/paper/1909.10594/) (2019)

## Built on by (most cited first, 30 of 98)

- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [Do Membership Inference Attacks Work on Large Language Models?](https://mlsanity.com/paper/2402.07841/) (2024)
- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies](https://mlsanity.com/paper/2407.19354/) (2024)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models](https://mlsanity.com/paper/2404.02936/) (2024)
- [ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods](https://mlsanity.com/paper/2406.15968/) (2024)
- [Dated Data: Tracing Knowledge Cutoffs in Large Language Models](https://mlsanity.com/paper/2403.12958/) (2024)
- [Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)](https://mlsanity.com/paper/2407.14937/) (2024)
- [Membership Inference Attacks Against In-Context Learning](https://mlsanity.com/paper/2409.01380/) (2024)
- [LLM-PBE: Assessing Data Privacy in Large Language Models](https://mlsanity.com/paper/2408.12787/) (2024)
- [Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applications](https://mlsanity.com/paper/2411.14110/) (2024)
- [Towards Label-Only Membership Inference Attack against Pre-trained Large Language Models](https://mlsanity.com/paper/2502.18943/) (2025)
- [Membership Inference Attacks against Large Vision-Language Models](https://mlsanity.com/paper/2411.02902/) (2024)
- [Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment](https://mlsanity.com/paper/2407.06443/) (2024)
- [Backdoored Retrievers for Prompt Injection Attacks on Retrieval Augmented Generation of Large Language Models](https://mlsanity.com/paper/2410.14479/) (2024)
- [A Survey on Privacy Risks and Protection in Large Language Models](https://mlsanity.com/paper/2505.01976/) (2025)
- [Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decoding](https://mlsanity.com/paper/2409.03363/) (2024)
- [MIA-Tuner: Adapting Large Language Models as Pre-training Text Detector](https://mlsanity.com/paper/2408.08661/) (2024)
- [Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directions](https://mlsanity.com/paper/2412.16504/) (2024)
- [Malicious LLM-Based Conversational AI Makes Users Reveal Personal Information](https://mlsanity.com/paper/2506.11680/) (2025)
- [A Comprehensive Survey of Contamination Detection Methods in Large Language Models](https://mlsanity.com/paper/2404.00699/) (2024)
- [Unveiling Over-Memorization in Finetuning LLMs for Reasoning Tasks](https://mlsanity.com/paper/2508.04117/) (2025)
- [A Survey on Responsible Generative AI: What to Generate and What Not](https://mlsanity.com/paper/2404.05783/) (2024)
- [A Middle Path for On-Premises LLM Deployment: Preserving Privacy Without Sacrificing Model Confidentiality](https://mlsanity.com/paper/2410.11182/) (2024)
- [Auditing Data Membership in Reinforcement Learning With Verifiable Rewards](https://mlsanity.com/paper/2511.14045/) (2025)
- [Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models](https://mlsanity.com/paper/2510.09259/) (2025)
- [G-Drift MIA: Membership Inference via Gradient-Induced Feature Drift in LLMs](https://mlsanity.com/paper/2604.00419/) (2026)
- [Unlocking Post-hoc Dataset Inference with Synthetic Data](https://mlsanity.com/paper/2506.15271/) (2025)

## Abstract (arXiv)

Membership Inference Attacks (MIA) aim to infer whether a target data record has been utilized for model training or not. Existing MIAs designed for large language models (LLMs) can be bifurcated into two types: reference-free and reference-based attacks. Although reference-based attacks appear promising performance by calibrating the probability measured on the target model with reference models, this illusion of privacy risk heavily depends on a reference dataset that closely resembles the training set. Both two types of attacks are predicated on the hypothesis that training records consistently maintain a higher probability of being sampled. However, this hypothesis heavily relies on the overfitting of target models, which will be mitigated by multiple regularization methods and the generalization of LLMs. Thus, these reasons lead to high false-positive rates of MIAs in practical scenarios. We propose a Membership Inference Attack based on Self-calibrated Probabilistic Variation (SPV-MIA). Specifically, we introduce a self-prompt approach, which constructs the dataset to fine-tune the reference model by prompting the target LLM itself. In this manner, the adversary can collect a dataset with a similar distribution from public APIs. Furthermore, we introduce probabilistic variation, a more reliable membership signal based on LLM memorization rather than overfitting, from which we rediscover the neighbour attack with theoretical grounding. Comprehensive evaluation conducted on three datasets and four exemplary LLMs shows that SPV-MIA raises the AUC of MIAs from 0.7 to a significantly high level of 0.9. Our code and dataset are available at: https://github.com/tsinghua-fib-lab/NeurIPS2024_SPV-MIA