# Information-Theoretic Privacy Control for Sequential Multi-Agent LLM Systems

arXiv: https://arxiv.org/abs/2603.05520 · HTML: https://mlsanity.com/paper/2603.05520/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 33 references that are in the map.
- First posted on arXiv 13 Feb 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 25 of 25)

- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases](https://mlsanity.com/paper/2407.12784/) (2024)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents](https://mlsanity.com/paper/2306.03314/) (2023)
- [TradingAgents: Multi-Agents LLM Financial Trading Framework](https://mlsanity.com/paper/2412.20138/) (2024)
- [PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action](https://mlsanity.com/paper/2409.00138/) (2024)
- [APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay](https://mlsanity.com/paper/2504.03601/) (2025)
- [AirGapAgent: Protecting Privacy-Conscious Conversational Agents](https://mlsanity.com/paper/2405.05175/) (2024)
- [AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2504.00587/) (2025)
- [MemoryGraft: Persistent Compromise of LLM Agents via Poisoned Experience Retrieval](https://mlsanity.com/paper/2512.16962/) (2025)
- [SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)](https://mlsanity.com/paper/2406.17975/) (2024)
- [Towards Enterprise-Ready Computer Using Generalist Agent](https://mlsanity.com/paper/2503.01861/) (2025)
- [Large Language Models are Advanced Anonymizers](https://mlsanity.com/paper/2402.13846/) (2024)
- [The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaboration](https://mlsanity.com/paper/2509.14284/) (2025)
- [1-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoning](https://mlsanity.com/paper/2508.07667/) (2025)
- [ConVerse: Benchmarking Contextual Safety in Agent-to-Agent Conversations](https://mlsanity.com/paper/2511.05359/) (2025)
- [Unifying Re-Identification, Attribute Inference, and Data Reconstruction Risks in Differential Privacy](https://mlsanity.com/paper/2507.06969/) (2025)
- [AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvement](https://mlsanity.com/paper/2502.00757/) (2025)
- [Positional Fragility in LLMs: How Offset Effects Reshape Our Understanding of Memorization Risks](https://mlsanity.com/paper/2505.13171/) (2025)
- [What Really is a Member? Discrediting Membership Inference via Poisoning](https://mlsanity.com/paper/2506.06003/) (2025)
- [Effective Policy Learning for Multi-Agent Online Coordination Beyond Submodular Objectives](https://mlsanity.com/paper/2509.22596/) (2025)
- [Generative AI Agents for Controllable and Protected Content Creation](https://mlsanity.com/paper/2601.12348/) (2026)

## Built on by (most cited first, 8 of 8)

- [Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation](https://mlsanity.com/paper/2606.10749/) (2026)
- [LCGuard: Latent Communication Guard for Safe KV Sharing in Multi-Agent Systems](https://mlsanity.com/paper/2605.22786/) (2026)
- [MCPHunt: An Evaluation Framework for Cross-Boundary Data Propagation in Multi-Server MCP Agents](https://mlsanity.com/paper/2604.27819/) (2026)
- [TRACE: Trajectory Risk-Aware Compression for Long-Horizon Agent Safety](https://mlsanity.com/paper/2606.00611/) (2026)
- [OCELOT: Inference-Leakage Budgets for Privacy-Preserving LLM Agents](https://mlsanity.com/paper/2606.12341/) (2026)
- [Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agents](https://mlsanity.com/paper/2606.26627/) (2026)
- [Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving](https://mlsanity.com/paper/2608.03579/) (2026)
- [BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agents](https://mlsanity.com/paper/2609.16305/) (2026)

## Abstract (arXiv)

Sequential multi-agent large language model (LLM) systems are increasingly deployed in sensitive domains such as healthcare, finance, and enterprise decision-making, where multiple specialized agents collaboratively process a single user request. Although individual agents may satisfy local privacy constraints, sensitive information can still be inferred through sequential composition and intermediate representations. In this work, we study compositional privacy leakage in sequential LLM agent pipelines. We formalize leakage using mutual information and derive a theoretical bound that characterizes how locally introduced leakage can amplify across agents under sequential execution. Motivated by this analysis, we propose a privacy-regularized training framework that directly constrains information flow between agent outputs and agent-local sensitive variables. We evaluate our approach across sequential agent pipelines of varying depth on three benchmark datasets, demonstrating stable optimization dynamics and consistent, interpretable privacy-utility trade-offs. Our results show that privacy in agentic LLM systems cannot be guaranteed by local constraints alone and must instead be treated as a system-level property during both training and deployment.