# Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers

arXiv: https://arxiv.org/abs/2506.15674 · HTML: https://mlsanity.com/paper/2506.15674/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 32 of its 53 references that are in the map.
- First posted on arXiv 18 Jun 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 30 of 32)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [Will we run out of data? Limits of LLM scaling based on human-generated data](https://mlsanity.com/paper/2211.04325/) (2022)
- [Reasoning Models Can Be Effective Without Thinking](https://mlsanity.com/paper/2504.09858/) (2025)
- [Phi-4-reasoning Technical Report](https://mlsanity.com/paper/2504.21318/) (2025)
- [SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities](https://mlsanity.com/paper/2502.12025/) (2025)
- [Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable](https://mlsanity.com/paper/2503.00555/) (2025)
- [The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1](https://mlsanity.com/paper/2502.12659/) (2025)
- [System 2 Attention (is something you might need too)](https://mlsanity.com/paper/2311.11829/) (2023)
- [AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents](https://mlsanity.com/paper/2503.09780/) (2025)
- [Effectively Controlling Reasoning Models through Thinking Intervention](https://mlsanity.com/paper/2503.24370/) (2025)
- [Safety in Large Reasoning Models: A Survey](https://mlsanity.com/paper/2504.17704/) (2025)
- [PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensembles](https://mlsanity.com/paper/2410.17127/) (2024)
- [Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math](https://mlsanity.com/paper/2504.21233/) (2025)
- [A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoning](https://mlsanity.com/paper/2501.02497/) (2025)
- [CI-Bench: Benchmarking Contextual Integrity of AI Assistants on Synthetic Data](https://mlsanity.com/paper/2409.13903/) (2024)
- [PrivaCI-Bench: Evaluating Privacy with Contextual Integrity and Legal Compliance](https://mlsanity.com/paper/2502.17041/) (2025)
- [To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts](https://mlsanity.com/paper/2410.14675/) (2024)
- [Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuning](https://mlsanity.com/paper/2502.09673/) (2025)
- [Position: Contextual Integrity is Inadequately Applied to Language Models](https://mlsanity.com/paper/2501.19173/) (2025)
- [Prompt Obfuscation for Large Language Models](https://mlsanity.com/paper/2409.11026/) (2024)

## Built on by (most cited first, 28 of 28)

- [A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasures](https://mlsanity.com/paper/2506.19676/) (2025)
- [A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models](https://mlsanity.com/paper/2509.03871/) (2025)
- [AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems](https://mlsanity.com/paper/2602.11510/) (2026)
- [Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2509.21882/) (2025)
- [SALT: Steering Activations towards Leakage-free Thinking in Chain of Thought](https://mlsanity.com/paper/2511.07772/) (2025)
- [Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation](https://mlsanity.com/paper/2512.16310/) (2025)
- [PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference Training](https://mlsanity.com/paper/2602.13840/) (2026)
- [Beyond Data Privacy: New Privacy Risks for Large Language Models](https://mlsanity.com/paper/2509.14278/) (2025)
- [When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Models](https://mlsanity.com/paper/2510.21285/) (2025)
- [Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models](https://mlsanity.com/paper/2601.15220/) (2026)
- [Chain-of-Sanitized-Thoughts: Plugging PII Leakage in CoT of Large Reasoning Models](https://mlsanity.com/paper/2601.05076/) (2026)
- [SPILLage: Agentic Oversharing on the Web](https://mlsanity.com/paper/2602.13516/) (2026)
- [RedAct: Redacting Agent Capability Traces for Procedural Skill Protection](https://mlsanity.com/paper/2606.10813/) (2026)
- [MuPPET: A Benchmark for Contextual Privacy of LLM Assistants in Multi-Party Conversations](https://mlsanity.com/paper/2606.23217/) (2026)
- [Does More Inference-Time Compute Really Help Robustness?](https://mlsanity.com/paper/2507.15974/) (2025)
- [Multi-Agent Framework for Threat Mitigation and Resilience in AI-Based Systems](https://mlsanity.com/paper/2512.23132/) (2025)
- [When Reasoning Leaks Membership: Membership Inference Attack on Black-box Large Reasoning Models](https://mlsanity.com/paper/2601.13607/) (2026)
- [Minim: Privacy-Aware Minimal View for Agents via Trusted Local Sanitization](https://mlsanity.com/paper/2606.13949/) (2026)
- [PrivacyAlign: Contextual Privacy Alignment for LLM Agents](https://mlsanity.com/paper/2606.21710/) (2026)
- [Black Box Absorption: LLMs Undermining Innovative Ideas](https://mlsanity.com/paper/2510.20612/) (2025)
- [FERA: Uncertainty-Aware Federated Reasoning for Large Language Models](https://mlsanity.com/paper/2605.10082/) (2026)
- [A Primer in Post-Training Reasoning Data: What We Know About How It Works](https://mlsanity.com/paper/2606.02113/) (2026)
- [ReasoningLens: Hierarchical Visualization and Diagnostic Auditing for Large Reasoning Models](https://mlsanity.com/paper/2606.23404/) (2026)
- [The Natural Language Interaction Protocol and Standard for AI Agents](https://mlsanity.com/paper/2609.04135/) (2026)
- [Personalized Privacy Control in LLMs via Attention Head Intervention](https://mlsanity.com/paper/2608.21209/) (2026)
- ["Nothing to See Here'': Unintended Disclosure through Revision Traces of LLM Deliverables](https://mlsanity.com/paper/2609.35408/) (2026)
- [Hindsight-Guided Rationale Distillation for Rare Disease Diagnosis](https://mlsanity.com/paper/2610.03176/) (2026)
- [Loose Lips: Understanding Privacy Leakage in Agentic Dialogue](https://mlsanity.com/paper/2610.14071/) (2026)

## Abstract (arXiv)

We study privacy leakage in the reasoning traces of large reasoning models used as personal agents. Unlike final outputs, reasoning traces are often assumed to be internal and safe. We challenge this assumption by showing that reasoning traces frequently contain sensitive user data, which can be extracted via prompt injections or accidentally leak into outputs. Through probing and agentic evaluations, we demonstrate that test-time compute approaches, particularly increased reasoning steps, amplify such leakage. While increasing the budget of those test-time compute approaches makes models more cautious in their final answers, it also leads them to reason more verbosely and leak more in their own thinking. This reveals a core tension: reasoning improves utility but enlarges the privacy attack surface. We argue that safety efforts must extend to the model's internal thinking, not just its outputs.