Has My System Prompt Been Used? Large Language Model Prompt Membership Inference
On the map: LLM safety and security → Prompt injection and agent security
Abstract (arXiv)
Prompt engineering has emerged as a powerful technique for optimizing large language models (LLMs) for specific applications, enabling faster prototyping and improved performance, and giving rise to the interest of the community in protecting proprietary system prompts. In this work, we explore a novel perspective on prompt privacy through the lens of membership inference. We develop Prompt Detective, a statistical method to reliably determine whether a given system prompt was used by a third-party language model. Our approach relies on a statistical test comparing the distributions of two groups of model outputs corresponding to different system prompts. Through extensive experiments with a variety of language models, we demonstrate the effectiveness of Prompt Detective for prompt membership inference. Our work reveals that even minor changes in system prompts manifest in distinct response distributions, enabling us to verify prompt usage with statistical significance.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 46 of its 56 references that are in the map. First posted on arXiv 14 Feb 2025.
Builds on
46 of 56 references.
References · 46
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023GPT-4 Technical Reportcited by 18,132
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023Mistral 7Bcited by 3,065
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
- 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
- 2024Mixtral of Expertscited by 1,785
34 more
- 2022Language Models (Mostly) Know What They Knowcited by 1,783
- 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
- 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
- 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
- 2021Membership Inference Attacks From First Principlescited by 839
- 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
- 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 475
- 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
- 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 317
- 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
- 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 273
- 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 265
- 2024Many-Shot In-Context Learningcited by 203
- 2021How Many Data Points is a Prompt Worth?cited by 197
- 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
- 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
- 2023Take a Step Back: Evoking Reasoning via Abstraction in Large Language Modelscited by 157
- 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 143
- 2023Are Diffusion Models Vulnerable to Membership Inference Attacks?cited by 135
- 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
- 2021Dataset Inference: Ownership Resolution in Machine Learningcited by 115
- 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 113
- 2023Large Language Models as Analogical Reasonerscited by 107
- 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
- 2023Membership Inference Attacks against Diffusion Modelscited by 84
- 2023Effective Prompt Extraction from Language Modelscited by 83
- 2023Thread of Thought Unraveling Chaotic Contextscited by 71
- 2023Language Model Inversioncited by 66
- 2023Scalable Membership Inference Attacks via Quantile Regressioncited by 61
- 2023RCOT: Detecting and Rectifying Factual Inconsistency in Reasoning by Reversing Chain-of-Thoughtcited by 47
- 2024Prompt Stealing Attacks Against Large Language Modelscited by 39
- 2022Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queriescited by 36
- 2023Explanation Selection Using Unlabeled Data for Chain-of-Thought Promptingcited by 12
- 2024Talking Nonsense: Probing Large Language Models' Understanding of Adversarial Gibberish Inputscited by 11
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.