# Embedding Information in Policy Randomness: Distribution-Preserving Implicit Communication in MDPs

arXiv: https://arxiv.org/abs/2610.15976 · HTML: https://mlsanity.com/paper/2610.15976/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 11 Oct 2026.

## Builds on (most cited first, 20 of 20)

- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [Understanding the impact of entropy on policy optimization](https://mlsanity.com/paper/1811.11214/) (2018)
- [Maximum Entropy RL (Provably) Solves Some Robust RL Problems](https://mlsanity.com/paper/2103.06257/) (2021)
- [Deep Reinforcement Learning for Cyber Security](https://mlsanity.com/paper/1906.05799/) (2019)
- [Advancing Beyond Identification: Multi-bit Watermark for Large Language Models](https://mlsanity.com/paper/2308.00221/) (2023)
- [Towards Codable Watermarking for Injecting Multi-bits Information to LLMs](https://mlsanity.com/paper/2307.15992/) (2023)
- [Maximum Entropy-Regularized Multi-Goal Reinforcement Learning](https://mlsanity.com/paper/1905.08786/) (2019)
- [A Review of Deep Reinforcement Learning for Smart Building Energy Management](https://mlsanity.com/paper/2008.05074/) (2020)
- [Communicating via Markov Decision Processes](https://mlsanity.com/paper/2107.08295/) (2021)
- [Agent Guide: A Simple Agent Behavioral Watermarking Framework](https://mlsanity.com/paper/2504.05871/) (2025)
- [Sequential Triggers for Watermarking of Deep Reinforcement Learning Policies](https://mlsanity.com/paper/1906.01126/) (2019)
- [WaterMod: Modular Token-Rank Partitioning for Probability-Balanced LLM Watermarking](https://mlsanity.com/paper/2511.07863/) (2025)
- [Sequential Behavioral Watermarking for LLM Agents](https://mlsanity.com/paper/2605.11036/) (2026)
- [TRACE: A Two-Channel Robust Attribution Watermark via Complementary Embeddings for LLM-Agent Trajectories](https://mlsanity.com/paper/2607.08400/) (2026)
- [Least Inferable Policies for Markov Decision Processes](https://mlsanity.com/paper/1809.06482/) (2018)
- [QuantileMark: A Message-Symmetric Multi-bit Watermark for LLMs](https://mlsanity.com/paper/2604.13786/) (2026)
- [Reinforcement Learning with Intrinsically Motivated Feedback Graph for Lost-sales Inventory Control](https://mlsanity.com/paper/2406.18351/) (2024)
- [Remotely Detectable Robot Policy Watermarking](https://mlsanity.com/paper/2512.15379/) (2025)
- [Communication over LQG Control Systems: A Convex Optimization Approach to Capacity](https://mlsanity.com/paper/2509.17002/) (2025)
- [The Control Plant as A Communication Channel: Implicit Communication for Decentralized LQG Control](https://mlsanity.com/paper/2605.01903/) (2026)

## Abstract (arXiv)

This paper investigates a new role of randomness in stochastic policies--as a medium for embedding and communicating information. Specifically, we consider an agent interacting with an environment according to a prescribed stochastic policy. By carefully modulating the random action realizations of the policy, information can be embedded into the agent's behavior without altering the prescribed action distribution. We cast this problem as a channel coding problem, in which the controlled system implicitly serves as a communication channel. We then propose a coding scheme, termed posterior-matching randomness modulation (PRAM), that selects actions according to the message to be transmitted. With the aid of common randomness independent of the control system, PRAM preserves the prescribed action distribution for every state and every message. We further prove that PRAM is asymptotically reliable: the message can be decoded with arbitrarily small error probability as the observed state trajectory becomes sufficiently long. Experiments demonstrate the effectiveness of the proposed framework in applications including multi-bit policy watermarking and trajectory provenance.