mlsanityData to 14 Oct 2026

Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers

arXiv 2506.15674 ↗First posted 18 Jun 2025cs.CL cs.AI cs.CRcited by 28

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

We study privacy leakage in the reasoning traces of large reasoning models used as personal agents. Unlike final outputs, reasoning traces are often assumed to be internal and safe. We challenge this assumption by showing that reasoning traces frequently contain sensitive user data, which can be extracted via prompt injections or accidentally leak into outputs. Through probing and agentic evaluations, we demonstrate that test-time compute approaches, particularly increased reasoning steps, amplify such leakage. While increasing the budget of those test-time compute approaches makes models more cautious in their final answers, it also leads them to reason more verbosely and leak more in their own thinking. This reveals a core tension: reasoning improves utility but enlarges the privacy attack surface. We argue that safety efforts must extend to the model's internal thinking, not just its outputs.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 32 of its 53 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

32 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Safety in Large Reasoning Models: A Survey24 Apr
  2. 2025Exploring the Necessity of Reasoning in LLM-based Agent Scenarios14 Mar
  3. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable1 Mar
  4. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R118 Feb
  5. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities17 Feb
  6. 2025Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuning13 Feb

Other references · 26

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  6. 2025s1: Simple test-time scalingcited by 1,341
  7. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  8. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  9. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  10. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  11. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  12. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
14 more
  1. 2025Reasoning Models Can Be Effective Without Thinkingcited by 187
  2. 2025Phi-4-reasoning Technical Reportcited by 151
  3. 2023System 2 Attention (is something you might need too)cited by 89
  4. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agentscited by 82
  5. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 65
  6. 2024PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensemblescited by 57
  7. 2025Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Mathcited by 44
  8. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  9. 2024CI-Bench: Benchmarking Contextual Integrity of AI Assistants on Synthetic Datacited by 37
  10. 2025PrivaCI-Bench: Evaluating Privacy with Contextual Integrity and Legal Compliancecited by 37
  11. 2024To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contextscited by 25
  12. 2025Position: Contextual Integrity is Inadequately Applied to Language Modelscited by 11
  13. 2024Prompt Obfuscation for Large Language Modelscited by 10
  14. 2024Whispers in the Machine: Confidentiality in Agentic Systemscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 18.

28 papers.

Sort:

By area:

  1. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 57
  2. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
  3. 2026AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systemscited by 25
  4. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  5. 2025SALT: Steering Activations towards Leakage-free Thinking in Chain of Thoughtcited by 11
  6. 2025Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigationcited by 11
  7. 2026PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference Trainingcited by 7
  8. 2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
  9. 2025When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Modelscited by 6
  10. 2026Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Modelscited by 6
  11. 2026Chain-of-Sanitized-Thoughts: Plugging PII Leakage in CoT of Large Reasoning Modelscited by 5
  12. 2026SPILLage: Agentic Oversharing on the Webcited by 3
16 more
  1. 2026RedAct: Redacting Agent Capability Traces for Procedural Skill Protectioncited by 3
  2. 2026MuPPET: A Benchmark for Contextual Privacy of LLM Assistants in Multi-Party Conversationscited by 3
  3. 2025Does More Inference-Time Compute Really Help Robustness?cited by 2
  4. 2025Multi-Agent Framework for Threat Mitigation and Resilience in AI-Based Systemscited by 2
  5. 2026When Reasoning Leaks Membership: Membership Inference Attack on Black-box Large Reasoning Modelscited by 2
  6. 2026Minim: Privacy-Aware Minimal View for Agents via Trusted Local Sanitizationcited by 2
  7. 2026PrivacyAlign: Contextual Privacy Alignment for LLM Agentscited by 2
  8. 2025Black Box Absorption: LLMs Undermining Innovative Ideascited by 1
  9. 2026FERA: Uncertainty-Aware Federated Reasoning for Large Language Modelscited by 1
  10. 2026A Primer in Post-Training Reasoning Data: What We Know About How It Workscited by 1
  11. 2026ReasoningLens: Hierarchical Visualization and Diagnostic Auditing for Large Reasoning Modelscited by 1
  12. 2026The Natural Language Interaction Protocol and Standard for AI Agentscited by 1
  13. 2026Personalized Privacy Control in LLMs via Attention Head Interventioncited by 0
  14. 2026"Nothing to See Here'': Unintended Disclosure through Revision Traces of LLM Deliverablescited by 0
  15. 2026Hindsight-Guided Rationale Distillation for Rare Disease Diagnosiscited by 0
  16. 2026Loose Lips: Understanding Privacy Leakage in Agentic Dialoguecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Leaky Thoughts to Private Reasoning: Controlling What LRMs Say to Themselves
  2. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
  3. 2026Safer Reasoning Traces: Measuring and Mitigating Chain-of-Thought Leakage in LLMs
  4. 2026OCELOT: Inference-Leakage Budgets for Privacy-Preserving LLM Agents
  5. 2025Personalizing Agent Privacy Decisions via Logical Entailment
  6. 2025Beyond Verification: Abductive Explanations for Post-AI Assessment of Privacy Leakage
  7. 2026Hidden Reasoning Must Leak, but Need Not Be Readable: Fundamental Opportunities and Limits for Chain-of-Thought Monitoring
  8. 2024Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent