# δ-STEAL: LLM Stealing Attack with Local Differential Privacy

arXiv: https://arxiv.org/abs/2510.21946 · HTML: https://mlsanity.com/paper/2510.21946/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security.
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 24 Oct 2025.

## Placed in

- LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/)

## Builds on (most cited first, 19 of 19)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbations](https://mlsanity.com/paper/1910.08902/) (2019)
- [MarkLLM: An Open-Source Toolkit for LLM Watermarking](https://mlsanity.com/paper/2405.10051/) (2024)
- [Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models](https://mlsanity.com/paper/2311.04378/) (2023)
- [Protecting Intellectual Property of Language Generation APIs with Lexical Watermark](https://mlsanity.com/paper/2112.02701/) (2021)
- [Imitation Attacks and Defenses for Black-box Machine Translation Systems](https://mlsanity.com/paper/2004.15015/) (2020)
- [On the Learnability of Watermarks for Language Models](https://mlsanity.com/paper/2312.04469/) (2023)
- [CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarks](https://mlsanity.com/paper/2209.08773/) (2022)
- [Watermarking Text Generated by Black-Box Language Models](https://mlsanity.com/paper/2305.08883/) (2023)
- [WaterMax: breaking the LLM watermark detectability-robustness-quality trade-off](https://mlsanity.com/paper/2403.04808/) (2024)
- [Watermarking Pre-trained Language Models with Backdooring](https://mlsanity.com/paper/2210.07543/) (2022)
- [Student Surpasses Teacher: Imitation Attack for Black-Box NLP APIs](https://mlsanity.com/paper/2108.13873/) (2021)
- [AdvParams: An Active DNN Intellectual Property Protection Technique via Adversarial Perturbation Based Parameter Encryption](https://mlsanity.com/paper/2105.13697/) (2021)
- [A Comparative Analysis of Word-Level Metric Differential Privacy: Benchmarking The Privacy-Utility Trade-off](https://mlsanity.com/paper/2404.03324/) (2024)
- [Towards Document-Level Paraphrase Generation with Sentence Rewriting and Reordering](https://mlsanity.com/paper/2109.07095/) (2021)
- [Stylometric Watermarks for Large Language Models](https://mlsanity.com/paper/2405.08400/) (2024)

## Built on by (most cited first, 3 of 3)

- [LLM-enabled Applications Require System-Level Threat Monitoring](https://mlsanity.com/paper/2602.19844/) (2026)
- [Robust LLM Watermarking with Minimal Semantic Distortion for IP Protection](https://mlsanity.com/paper/2605.23175/) (2026)
- [LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization](https://mlsanity.com/paper/2607.10103/) (2026)

## Abstract (arXiv)

Large language models (LLMs) demonstrate remarkable capabilities across various tasks. However, their deployment introduces significant risks related to intellectual property. In this context, we focus on model stealing attacks, where adversaries replicate the behaviors of these models to steal services. These attacks are highly relevant to proprietary LLMs and pose serious threats to revenue and financial stability. To mitigate these risks, the watermarking solution embeds imperceptible patterns in LLM outputs, enabling model traceability and intellectual property verification. In this paper, we study the vulnerability of LLM service providers by introducing δ-STEAL, a novel model stealing attack that bypasses the service provider's watermark detectors while preserving the adversary's model utility. δ-STEAL injects noise into the token embeddings of the adversary's model during fine-tuning in a way that satisfies local differential privacy (LDP) guarantees. The adversary queries the service provider's model to collect outputs and form input-output training pairs. By applying LDP-preserving noise to these pairs, δ-STEAL obfuscates watermark signals, making it difficult for the service provider to determine whether its outputs were used, thereby preventing claims of model theft. Our experiments show that δ-STEAL with lightweight modifications achieves attack success rates of up to 96.95% without significantly compromising the adversary's model utility. The noise scale in LDP controls the trade-off between attack effectiveness and model utility. This poses a significant risk, as even robust watermarks can be bypassed, allowing adversaries to deceive watermark detectors and undermine current intellectual property protection methods.