# Large Language Models as Carriers of Hidden Messages

arXiv: https://arxiv.org/abs/2406.02481 · HTML: https://mlsanity.com/paper/2406.02481/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 21 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 16 of 16)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [Beyond Memorization: Violating Privacy Via Inference with Large Language Models](https://mlsanity.com/paper/2310.07298/) (2023)
- [Effective Prompt Extraction from Language Models](https://mlsanity.com/paper/2307.06865/) (2023)
- [Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities](https://mlsanity.com/paper/2308.12833/) (2023)
- [Neural Linguistic Steganography](https://mlsanity.com/paper/1909.01496/) (2019)
- [Publicly-Detectable Watermarking for Language Models](https://mlsanity.com/paper/2310.18491/) (2023)
- [Instructional Fingerprinting of Large Language Models](https://mlsanity.com/paper/2401.12255/) (2024)
- [Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models](https://mlsanity.com/paper/2403.04786/) (2024)
- [Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks](https://mlsanity.com/paper/2311.09060/) (2023)
- [Generative Text Steganography with Large Language Model](https://mlsanity.com/paper/2404.10229/) (2024)
- [Special Characters Attack: Toward Scalable Training Data Extraction From Large Language Models](https://mlsanity.com/paper/2405.05990/) (2024)

## Built on by (most cited first, 2 of 2)

- [What LLMs Think When You Don't Tell Them What to Think About?](https://mlsanity.com/paper/2602.01689/) (2026)
- [Synchronized Logit Steering: Real-world Steganography](https://mlsanity.com/paper/2608.14697/) (2026)

## Abstract (arXiv)

Simple fine-tuning can embed hidden text into large language models (LLMs), which is revealed only when triggered by a specific query. Applications include LLM fingerprinting, where a unique identifier is embedded to verify licensing compliance, and steganography, where the LLM carries hidden messages disclosed through a trigger query. Our work demonstrates that embedding hidden text via fine-tuning, although seemingly secure due to the vast number of potential triggers, is vulnerable to extraction through analysis of the LLM's output decoding process. We introduce an extraction attack called Unconditional Token Forcing (UTF), which iteratively feeds tokens from the LLM's vocabulary to reveal sequences with high token probabilities, indicating hidden text candidates. We also present Unconditional Token Forcing Confusion (UTFC), a defense paradigm that makes hidden text resistant to all known extraction attacks without degrading the general performance of LLMs compared to standard fine-tuning. UTFC has both benign (improving LLM fingerprinting) and malign applications (using LLMs to create covert communication channels).