# A Survey on Model Extraction Attacks and Defenses for Large Language Models

arXiv: https://arxiv.org/abs/2506.22521 · HTML: https://mlsanity.com/paper/2506.22521/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 83 references that are in the map.
- First posted on arXiv 26 Jun 2025.

## Builds on (most cited first, 30 of 47)

- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://mlsanity.com/paper/2205.12628/) (2022)
- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [Stealing Part of a Production Language Model](https://mlsanity.com/paper/2403.06634/) (2024)
- [Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks](https://mlsanity.com/paper/2309.17410/) (2023)
- [O1 Replication Journey: A Strategic Progress Report -- Part 1](https://mlsanity.com/paper/2410.18982/) (2024)
- [Thieves on Sesame Street! Model Extraction of BERT-based APIs](https://mlsanity.com/paper/1910.12366/) (2019)
- [PLeak: Prompt Leaking Attacks against Large Language Model Applications](https://mlsanity.com/paper/2405.06823/) (2024)
- [A Survey of Text Watermarking in the Era of Large Language Models](https://mlsanity.com/paper/2312.07913/) (2023)
- [On Protecting the Data Privacy of Large Language Models (LLMs): A Survey](https://mlsanity.com/paper/2403.05156/) (2024)
- [O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?](https://mlsanity.com/paper/2411.16489/) (2024)
- [Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation](https://mlsanity.com/paper/2406.20053/) (2024)
- [A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems](https://mlsanity.com/paper/2402.18649/) (2024)
- [CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarks](https://mlsanity.com/paper/2209.08773/) (2022)
- [NSML: Meet the MLaaS platform with a real-world case study](https://mlsanity.com/paper/1810.09957/) (2018)
- [Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!](https://mlsanity.com/paper/2103.10013/) (2021)
- [Prompt Stealing Attacks Against Text-to-Image Generation Models](https://mlsanity.com/paper/2302.09923/) (2023)
- [Extracting Prompts by Inverting LLM Outputs](https://mlsanity.com/paper/2405.15012/) (2024)
- [Model Extraction Warning in MLaaS Paradigm](https://mlsanity.com/paper/1711.07221/) (2017)
- [Chiron: Privacy-preserving Machine Learning as a Service](https://mlsanity.com/paper/1803.05961/) (2018)
- [Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation](https://mlsanity.com/paper/2307.04401/) (2023)
- [Text Revealer: Private Text Reconstruction via Model Inversion Attacks against Transformers](https://mlsanity.com/paper/2209.10505/) (2022)
- [Unveiling Memorization in Code Models](https://mlsanity.com/paper/2308.09932/) (2023)
- [Exploring Safety-Utility Trade-Offs in Personalized Language Models](https://mlsanity.com/paper/2406.11107/) (2024)

## Built on by (most cited first, 19 of 19)

- [Intelligent AI Delegation](https://mlsanity.com/paper/2602.11865/) (2026)
- [On Protecting Agentic Systems' Intellectual Property via Watermarking](https://mlsanity.com/paper/2602.08401/) (2026)
- [Memorization in Large Language Models in Medicine: Prevalence, Characteristics, and Implications](https://mlsanity.com/paper/2509.08604/) (2025)
- [CREDIT: Certified Ownership Verification of Deep Neural Networks Against Model Extraction Attacks](https://mlsanity.com/paper/2602.20419/) (2026)
- [Sequential Behavioral Watermarking for LLM Agents](https://mlsanity.com/paper/2605.11036/) (2026)
- [GraphIP-Bench: How Hard Is It to Steal a Graph Neural Network, and Can We Stop It?](https://mlsanity.com/paper/2605.12827/) (2026)
- [Intellectual Property in Graph-Based Machine Learning as a Service: Attacks and Defenses](https://mlsanity.com/paper/2508.19641/) (2025)
- [CITED: A Decision Boundary-Aware Signature for GNNs Towards Model Extraction Defense](https://mlsanity.com/paper/2602.20418/) (2026)
- [Safety, Security, and Cognitive Risks in World Models](https://mlsanity.com/paper/2604.01346/) (2026)
- [From Essence to Defense: Adaptive Semantic-aware Watermarking for Embedding-as-a-Service Copyright Protection](https://mlsanity.com/paper/2512.16439/) (2025)
- [Echoes within the Reasoning: Stealthy and Effective Watermarking via Chain of Thought](https://mlsanity.com/paper/2605.28890/) (2026)
- [DESIGN: Encrypted GNN Inference via Server-Side Input Graph Pruning](https://mlsanity.com/paper/2507.05649/) (2025)
- [What Does It Mean to Break a Distillation Defense?](https://mlsanity.com/paper/2606.25059/) (2026)
- [Robustness of LLM-enabled vehicle trajectory prediction under data security threats](https://mlsanity.com/paper/2511.13753/) (2025)
- [Can Subgraph Explanations Be Weaponized to Steal Graph Neural Networks?](https://mlsanity.com/paper/2605.30470/) (2026)
- [RECAST: Model Reconstruction via Counterfactual-Aware Wasserstein Geometry under Limited Data](https://mlsanity.com/paper/2606.27948/) (2026)
- [Black-Box Inference of LLM Architectural Properties with Restrictive API Access](https://mlsanity.com/paper/2607.01313/) (2026)
- [ADS-C: Antidistillation Sampling for Classification](https://mlsanity.com/paper/2607.15467/) (2026)
- [JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocols](https://mlsanity.com/paper/2608.26982/) (2026)

## Abstract (arXiv)

Model extraction attacks pose significant security threats to deployed language models, potentially compromising intellectual property and user privacy. This survey provides a comprehensive taxonomy of LLM-specific extraction attacks and defenses, categorizing attacks into functionality extraction, training data extraction, and prompt-targeted attacks. We analyze various attack methodologies including API-based knowledge distillation, direct querying, parameter recovery, and prompt stealing techniques that exploit transformer architectures. We then examine defense mechanisms organized into model protection, data privacy protection, and prompt-targeted strategies, evaluating their effectiveness across different deployment scenarios. We propose specialized metrics for evaluating both attack effectiveness and defense performance, addressing the specific challenges of generative language models. Through our analysis, we identify critical limitations in current approaches and propose promising research directions, including integrated attack methodologies and adaptive defense mechanisms that balance security with model utility. This work serves NLP researchers, ML engineers, and security professionals seeking to protect language models in production environments.