# Addressing Tokenization Inconsistency in Steganography and Watermarking Based on Large Language Models

arXiv: https://arxiv.org/abs/2508.20718 · HTML: https://mlsanity.com/paper/2508.20718/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security.
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 28 Aug 2025.

## Placed in

- LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/)

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Robust Distortion-free Watermarks for Language Models](https://mlsanity.com/paper/2307.15593/) (2023)
- [Provable Robust Watermarking for AI-Generated Text](https://mlsanity.com/paper/2306.17439/) (2023)
- [Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities](https://mlsanity.com/paper/2404.17790/) (2024)
- [Coercing LLMs to do and reveal (almost) anything](https://mlsanity.com/paper/2402.14020/) (2024)
- [Neural Linguistic Steganography](https://mlsanity.com/paper/1909.01496/) (2019)
- [An Entropy-based Text Watermarking Detection Method](https://mlsanity.com/paper/2403.13485/) (2024)
- [The Threat of Offensive AI to Organizations](https://mlsanity.com/paper/2106.15764/) (2021)
- [GumbelSoft: Diversified Language Model Watermarking via the GumbelMax-trick](https://mlsanity.com/paper/2402.12948/) (2024)
- [Towards Near-imperceptible Steganographic Text](https://mlsanity.com/paper/1907.06679/) (2019)
- [Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization](https://mlsanity.com/paper/2405.17067/) (2024)
- [Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection](https://mlsanity.com/paper/2404.09894/) (2024)
- [Near-imperceptible Neural Linguistic Steganography via Self-Adjusting Arithmetic Coding](https://mlsanity.com/paper/2010.00677/) (2020)
- [Addressing Segmentation Ambiguity in Neural Linguistic Steganography](https://mlsanity.com/paper/2211.06662/) (2022)
- [Tokenization Consistency Matters for Generative Models on Extractive NLP Tasks](https://mlsanity.com/paper/2212.09912/) (2022)
- [GlitchMiner: Mining Glitch Tokens in Large Language Models via Gradient-based Discrete Optimization](https://mlsanity.com/paper/2410.15052/) (2024)

## Built on by (most cited first, 4 of 4)

- [Hide and Seek in Embedding Space: Geometry-based Steganography and Detection in Large Language Models](https://mlsanity.com/paper/2601.22818/) (2026)
- [HiTMS: A High-Throughput Multi-Stream Linguistic Steganography Framework](https://mlsanity.com/paper/2607.23597/) (2026)
- [Text Steganography with Dynamic Codebook and Multimodal Large Language Model](https://mlsanity.com/paper/2604.20269/) (2026)
- [Tokenization-Invariant Watermarking: A Byte-Local Method for LLM Text](https://mlsanity.com/paper/2610.14378/) (2026)

## Abstract (arXiv)

Large language models have significantly enhanced the capacities and efficiency of text generation. On the one hand, they have improved the quality of text-based steganography. On the other hand, they have also underscored the importance of watermarking as a safeguard against malicious misuse. In this study, we focus on tokenization inconsistency (TI) between Alice and Bob in steganography and watermarking, where TI can undermine robustness. Our investigation reveals that the problematic tokens responsible for TI exhibit two key characteristics: infrequency and temporariness. Based on these findings, we propose two tailored solutions for TI elimination: a stepwise verification method for steganography and a post-hoc rollback method for watermarking. Experiments show that (1) compared to traditional disambiguation methods in steganography, directly addressing TI leads to improvements in fluency, imperceptibility, and anti-steganalysis capacity; (2) for watermarking, addressing TI enhances detectability and robustness against attacks.