# Differential Privacy in Generative AI Agents: Analysis and Optimal Tradeoffs

arXiv: https://arxiv.org/abs/2603.17902 · HTML: https://mlsanity.com/paper/2603.17902/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 18 Mar 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 12 of 12)

- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies](https://mlsanity.com/paper/2407.19354/) (2024)
- [On Protecting the Data Privacy of Large Language Models (LLMs): A Survey](https://mlsanity.com/paper/2403.05156/) (2024)
- [Differential Privacy for Text Analytics via Natural Text Sanitization](https://mlsanity.com/paper/2106.01221/) (2021)
- [Privacy-Preserving Retrieval-Augmented Generation with Differential Privacy](https://mlsanity.com/paper/2412.04697/) (2024)
- [Differentially Private Decoding in Large Language Models](https://mlsanity.com/paper/2205.13621/) (2022)
- [Securing Agentic AI: A Comprehensive Threat Model and Mitigation Framework for Generative AI Agents](https://mlsanity.com/paper/2504.19956/) (2025)
- [Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trends](https://mlsanity.com/paper/2409.14457/) (2024)
- [Security of Internet of Agents: Attacks and Countermeasures](https://mlsanity.com/paper/2505.08807/) (2025)
- [DP-Fusion: Token-Level Differentially Private Inference for Large Language Models](https://mlsanity.com/paper/2507.04531/) (2025)
- [PrivCode: When Code Generation Meets Differential Privacy](https://mlsanity.com/paper/2512.05459/) (2025)
- [Differentially Private Distributed Nonconvex Stochastic Optimization with Quantized Communication](https://mlsanity.com/paper/2403.18254/) (2024)

## Built on by (most cited first, 1 of 1)

- [The Internet of Agentic AI: Communication, Coordination, and Collective Intelligence at Scale](https://mlsanity.com/paper/2606.12835/) (2026)

## Abstract (arXiv)

Large language models (LLMs) and AI agents are increasingly integrated into enterprise systems to access internal databases and generate context-aware responses. While such integration improves productivity and decision support, the model outputs may inadvertently reveal sensitive information. Although many prior efforts focus on protecting the privacy of user prompts, relatively few studies consider privacy risks from the enterprise data perspective. Hence, this paper develops a probabilistic framework for analyzing privacy leakage in AI agents based on differential privacy. We model response generation as a stochastic mechanism that maps prompts and datasets to distributions over token sequences. Within this framework, we introduce token-level and message-level differential privacy and derive privacy bounds that relate privacy leakage to generation parameters such as temperature and message length. We further formulate a privacy-utility design problem that characterizes optimal temperature selection.