# Privacy-R1: Privacy-Aware Multi-LLM Agent Collaboration via Reinforcement Learning

arXiv: https://arxiv.org/abs/2510.16054 · HTML: https://mlsanity.com/paper/2510.16054/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 16 Oct 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [MedDialog: Two Large-scale Medical Dialogue Datasets](https://mlsanity.com/paper/2004.03329/) (2020)
- [LLM Unlearning via Loss Adjustment with Only Forget Data](https://mlsanity.com/paper/2410.11143/) (2024)
- [PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensembles](https://mlsanity.com/paper/2410.17127/) (2024)
- [Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild](https://mlsanity.com/paper/2407.11438/) (2024)
- [Offline Reinforcement Learning with Differential Privacy](https://mlsanity.com/paper/2206.00810/) (2022)
- [ToxiCraft: A Novel Framework for Synthetic Generation of Harmful Information](https://mlsanity.com/paper/2409.14740/) (2024)
- [Defending Against Social Engineering Attacks in the Age of LLMs](https://mlsanity.com/paper/2406.12263/) (2024)
- [FRAMU: Attention-based Machine Unlearning using Federated Reinforcement Learning](https://mlsanity.com/paper/2309.10283/) (2023)
- [Toward Safe and Human-Aligned Game Conversational Recommendation via Multi-Agent Decomposition](https://mlsanity.com/paper/2504.20094/) (2025)
- [Private Memorization Editing: Turning Memorization into a Defense to Strengthen Data Privacy in Large Language Models](https://mlsanity.com/paper/2506.10024/) (2025)
- [Deep PackGen: A Deep Reinforcement Learning Framework for Adversarial Network Packet Generation](https://mlsanity.com/paper/2305.11039/) (2023)

## Built on by (most cited first, 6 of 6)

- [Value of Information: A Framework for Human-Agent Communication](https://mlsanity.com/paper/2601.06407/) (2026)
- [Steer Model beyond Assistant: Controlling System Prompt Strength via Contrastive Decoding](https://mlsanity.com/paper/2601.06403/) (2026)
- [Beyond Refusal: Probing the Limits of Agentic Self-Correction for Semantic Sensitive Information](https://mlsanity.com/paper/2602.21496/) (2026)
- [Lower-Resource, Higher Scores: Language Bias in LLM Evaluators](https://mlsanity.com/paper/2607.14480/) (2026)
- [PrivMeSA: Privacy-Aware Self-Evolving Multi-Agent System for Medicine via Local-Remote LLM Collaboration](https://mlsanity.com/paper/2609.38458/) (2026)
- [ReCast: Contract-Preserving Protection for Fixed-Interface Multimodal Reasoning](https://mlsanity.com/paper/2610.01184/) (2026)

## Abstract (arXiv)

When users submit queries to Large Language Models (LLMs), their prompts can often contain sensitive data, forcing a difficult choice: Send the query to a powerful proprietary LLM providers to achieving state-of-the-art performance and risk data exposure, or relying on smaller, local models guarantees data privacy but often results in a degradation of task performance. Prior approaches have relied on static pipelines that use LLM rewriting, which shatters linguistic coherence and indiscriminately removes privacy-sensitive information, including task-critical content. We reformulate this challenge (Privacy-Conscious Delegation) as a sequential decision-making problem and introduce a novel reinforcement learning (RL) framework called Privacy-R1 to solve it. Our framework trains an agent to dynamically route text chunks, learning a policy that optimally balances the trade-off between privacy leakage and task performance. It implicitly distinguishes between replaceable Personally Identifiable Information (PII) (which it shields locally) and task-critical PII (which it strategically sends to the remote model for maximal utility). To validate our approach in complex scenarios, we also introduce a new medical dataset with high PII density. Our framework achieves a new state-of-the-art on the privacy-utility frontier, demonstrating the necessity of learned, adaptive policies for deploying LLMs in sensitive environments. Dataset can be found at: https://github.com/zackhuiiiii/Privacy-R1.