# TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agent

arXiv: https://arxiv.org/abs/2505.20118 · HTML: https://mlsanity.com/paper/2505.20118/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 46 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 20 of 20)

- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Provable Robust Watermarking for AI-Generated Text](https://mlsanity.com/paper/2306.17439/) (2023)
- [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://mlsanity.com/paper/2310.17884/) (2023)
- [BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents](https://mlsanity.com/paper/2406.03007/) (2024)
- [Backdoor Attacks for In-Context Learning with Language Models](https://mlsanity.com/paper/2307.14692/) (2023)
- [Neural Linguistic Steganography](https://mlsanity.com/paper/1909.01496/) (2019)
- [Preventing Language Models From Hiding Their Reasoning](https://mlsanity.com/paper/2310.18512/) (2023)
- [Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMs](https://mlsanity.com/paper/2410.03768/) (2024)
- [Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)](https://mlsanity.com/paper/2407.14937/) (2024)
- [Generating Steganographic Text with LSTMs](https://mlsanity.com/paper/1705.10742/) (2017)
- [Perfectly Secure Steganography Using Minimum Entropy Coupling](https://mlsanity.com/paper/2210.14889/) (2022)
- [Stealthy and Persistent Unalignment on Large Language Models via Backdoor Injections](https://mlsanity.com/paper/2312.00027/) (2023)
- [Near-imperceptible Neural Linguistic Steganography via Self-Adjusting Arithmetic Coding](https://mlsanity.com/paper/2010.00677/) (2020)
- [Rag and Roll: An End-to-End Evaluation of Indirect Prompt Manipulations in LLM-based Application Frameworks](https://mlsanity.com/paper/2408.05025/) (2024)
- [AI Usage Cards: Responsibly Reporting AI-generated Content](https://mlsanity.com/paper/2303.03886/) (2023)
- [A Study of Backdoors in Instruction Fine-tuned Language Models](https://mlsanity.com/paper/2406.07778/) (2024)
- [CADS: A Systematic Literature Review on the Challenges of Abstractive Dialogue Summarization](https://mlsanity.com/paper/2406.07494/) (2024)
- [OD-Stega: LLM-Based Relatively Secure Steganography via Optimized Distributions](https://mlsanity.com/paper/2410.04328/) (2024)
- [Preventing Jailbreak Prompts as Malicious Tools for Cybercriminals: A Cyber Defense Perspective](https://mlsanity.com/paper/2411.16642/) (2024)
- [Leveraging Generative Models for Covert Messaging: Challenges and Tradeoffs for "Dead-Drop" Deployments](https://mlsanity.com/paper/2110.07009/) (2021)

## Built on by (most cited first, 9 of 9)

- [Hide and Seek in Embedding Space: Geometry-based Steganography and Detection in Large Language Models](https://mlsanity.com/paper/2601.22818/) (2026)
- [Steganography Without Modification: Hidden Communication via LLM Seeds](https://mlsanity.com/paper/2606.09135/) (2026)
- [Compositional Threat Analysis of Latent Compromise in LLM Agent Systems: The Order 66 Scenario](https://mlsanity.com/paper/2608.08131/) (2026)
- [BabelSteering: Multilingual Safety Alignment via English Steering Vectors](https://mlsanity.com/paper/2608.16577/) (2026)
- [Cordyceps: Covert Control Attacks on LLMs via Data Poisoning](https://mlsanity.com/paper/2605.26595/) (2026)
- [The Invitation Trap: Proactive Availability Backdoor in LLMs via Conversational Induction](https://mlsanity.com/paper/2606.00654/) (2026)
- [Now You (Still) See Me: Detecting Evasive Steganographic Payloads in LLMs](https://mlsanity.com/paper/2606.09411/) (2026)
- [Inadvertent Context Leakage in Language Models](https://mlsanity.com/paper/2608.19857/) (2026)
- [For Your Eyes Only: Evaluating Coordination Between Isolated Language Model Instances](https://mlsanity.com/paper/2609.19504/) (2026)

## Abstract (arXiv)

As large language models (LLMs) become integrated into sensitive workflows, concerns grow over their potential to leak confidential information. We propose TrojanStego, a novel threat model in which an adversary fine-tunes an LLM to embed sensitive context information into natural-looking outputs via linguistic steganography, without requiring explicit control over inference inputs. We introduce a taxonomy outlining risk factors for compromised LLMs, and use it to evaluate the risk profile of the threat. To implement TrojanStego, we propose a practical encoding scheme based on vocabulary partitioning learnable by LLMs via fine-tuning. Experimental results show that compromised models reliably transmit 32-bit secrets with 87% accuracy on held-out prompts, reaching over 97% accuracy using majority voting across three generations. Further, they maintain high utility, can evade human detection, and preserve coherence. These results highlight a new class of LLM data exfiltration attacks that are passive, covert, practical, and dangerous.