# Can You Keep a Secret? Involuntary Information Leakage in Language Model Writing

arXiv: https://arxiv.org/abs/2605.10794 · HTML: https://mlsanity.com/paper/2605.10794/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 11 May 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 10 of 10)

- [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://mlsanity.com/paper/2310.17884/) (2023)
- [Base Models Beat Aligned Models at Randomness and Creativity](https://mlsanity.com/paper/2505.00047/) (2025)
- [Eliciting Secret Knowledge from Language Models](https://mlsanity.com/paper/2510.01070/) (2025)
- [LLM Probability Concentration: How Alignment Shrinks the Generative Horizon](https://mlsanity.com/paper/2506.17871/) (2025)
- [Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models](https://mlsanity.com/paper/2408.06518/) (2024)
- [Do LLMs Strategically Reveal, Conceal, and Infer Information? A Theoretical and Empirical Analysis in The Chameleon Game](https://mlsanity.com/paper/2501.19398/) (2025)
- [LLMs Can't Play Hangman: On the Necessity of a Private Working Memory for Language Agents](https://mlsanity.com/paper/2601.06973/) (2026)
- [Has My System Prompt Been Used? Large Language Model Prompt Membership Inference](https://mlsanity.com/paper/2502.09974/) (2025)
- [Scaling Down Semantic Leakage: Investigating Associative Bias in Smaller Language Models](https://mlsanity.com/paper/2501.06638/) (2025)
- [Don't Think of the White Bear: Ironic Negation in Transformer Models Under Cognitive Load](https://mlsanity.com/paper/2511.12381/) (2025)

## Built on by (most cited first, 1 of 1)

- ["Nothing to See Here'': Unintended Disclosure through Revision Traces of LLM Deliverables](https://mlsanity.com/paper/2609.35408/) (2026)

## Abstract (arXiv)

Language models are deployed in settings that require compartmentalization: system prompts should not be disclosed, chain-of-thought reasoning is hidden from users, and sensitive data passes through shared contexts. We test whether models can keep prompted information out of their writing. We give each model a secret word with instructions not to reveal it, then ask it to write a story. A second model tries to identify the secret from the story in a binary discrimination test. The secret word never appears literally in any output, but all five frontier models we test leak it thematically -- through topic choice, imagery, and setting--6hy-at rates significantly different from chance, up to 79%. When told to actively hide the secret, models write away from it, and this avoidance is itself detectable. The leakage is cross-model readable, scales sharply with model size within two model families, and disappears entirely for short-form writing like jokes. Giving the model a decoy concept to ``focus on instead'' partially redirects the leakage from the real secret to the decoy. Attending to a secret appears to open up an information channel that frontier LLMs cannot close, even when instructed to.