# Autoregressive Linguistic Steganography Based on BERT and Consistency Coding

arXiv: https://arxiv.org/abs/2203.13972 · HTML: https://mlsanity.com/paper/2203.13972/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 34 references that are in the map.
- First posted on arXiv 26 Mar 2022.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Neural Linguistic Steganography](https://mlsanity.com/paper/1909.01496/) (2019)
- [Generating Steganographic Text with LSTMs](https://mlsanity.com/paper/1705.10742/) (2017)
- [Frustratingly Easy Edit-based Linguistic Steganography with a Masked Language Model](https://mlsanity.com/paper/2104.09833/) (2021)
- [Exploiting Language Model for Efficient Linguistic Steganalysis](https://mlsanity.com/paper/2107.12168/) (2021)

## Built on by (most cited first, 6 of 6)

- [Addressing Segmentation Ambiguity in Neural Linguistic Steganography](https://mlsanity.com/paper/2211.06662/) (2022)
- [General Framework for Reversible Data Hiding in Texts Based on Masked Language Modeling](https://mlsanity.com/paper/2206.10112/) (2022)
- [A Content-Preserving Secure Linguistic Steganography](https://mlsanity.com/paper/2511.12565/) (2025)
- [HiTMS: A High-Throughput Multi-Stream Linguistic Steganography Framework](https://mlsanity.com/paper/2607.23597/) (2026)
- [Yet Another Watermark for Large Language Models](https://mlsanity.com/paper/2509.12574/) (2025)
- [A Comprehensive Survey on Linguistic Steganography: Methods, Countermeasures, Evaluation, and Challenges](https://mlsanity.com/paper/2608.29077/) (2026)

## Abstract (arXiv)

Linguistic steganography (LS) conceals the presence of communication by embedding secret information into a text. How to generate a high-quality text carrying secret information is a key problem. With the widespread application of deep learning in natural language processing, recent algorithms use a language model (LM) to generate the steganographic text, which provides a higher payload compared with many previous arts. However, the security still needs to be enhanced. To tackle with this problem, we propose a novel autoregressive LS algorithm based on BERT and consistency coding, which achieves a better trade-off between embedding payload and system security. In the proposed work, based on the introduction of the masked LM, given a text, we use consistency coding to make up for the shortcomings of block coding used in the previous work so that we can encode arbitrary-size candidate token set and take advantages of the probability distribution for information hiding. The masked positions to be embedded are filled with tokens determined by an autoregressive manner to enhance the connection between contexts and therefore maintain the quality of the text. Experimental results have shown that, compared with related works, the proposed work improves the fluency of the steganographic text while guaranteeing security, and also increases the embedding payload to a certain extent.