Reinforcement Learning on Benign Facts Amplifies Leakage of Memorized Private Data
On the map: Deep reinforcement learning methods → RL training of language and agent models
Abstract (arXiv)
Reinforcement learning with verifiable rewards (RLVR) is deployed to make models better at reasoning tasks, but its side effect on what models will divulge is under studied. Here we show that RLVR on facts increases extraction of personally identifiable information (PII) the instruct model had already memorized. We first confirm that instruct models have already memorized PII but leave them latent, rarely surfacing one when asked. We then apply RL on benign factual data that contains no PII of any kind, and re-probe: a targeted probe over name->email pairs, and an untargeted free-recall prompt that simply asks the model to list the addresses it knows. PII extraction rises sharply under both: on DeepSeek-V3.1, verbatim recall@k increases from 0.155 to 0.370, a 2.4x gain. The effect scales with model size: across three models spanning 8B to 671B parameters, absolute leakage is largest in the biggest model. Meanwhile model's reasoning abilities and refusal rates are retained, indicating that RL selectively changes which memorized information is accessible rather than broadly altering the model. In summary, memorized private data can be made markedly more extractable by training that never touches it. This gives an adversary a route to memorized data that requires no privacy-relevant training signal and no access to the data itself -- only the ability to fine-tune on something innocuous.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 14 of its 27 references that are in the map. First posted on arXiv 22 Aug 2026.
Builds on
14 of 27 references.
References · 14
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2020Extracting Training Data from Large Language Modelscited by 2,189
- 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
- 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
- 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 285
- 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
- 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
- 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
- 2025Persona Features Control Emergent Misalignmentcited by 86
- 2023The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Riskscited by 36
- 2026Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMscited by 16
- 2026Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Modelscited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Unintended Memorization of Sensitive Information in Fine-Tuned Language Models
- 2026Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewards
- 2026Safer Reasoning Traces: Measuring and Mitigating Chain-of-Thought Leakage in LLMs
- 2026Deployment-Time Memorization in Foundation-Model Agents
- 2025Auditing Data Membership in Reinforcement Learning With Verifiable Rewards
- 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards
- 2026Discovering Universal Activation Directions for PII Leakage in Language Models
- 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward