# Information-Theoretic Opacity-Enforcement in Markov Decision Processes

arXiv: https://arxiv.org/abs/2405.00157 · HTML: https://mlsanity.com/paper/2405.00157/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 9 of its 32 references that are in the map.
- First posted on arXiv 30 Apr 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 9 of 9)

- [Bridging the Gap Between Value and Policy Based Reinforcement Learning](https://mlsanity.com/paper/1702.08892/) (2017)
- [Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization](https://mlsanity.com/paper/2007.06558/) (2020)
- [Entropy Maximization for Markov Decision Processes Under Temporal Logic Constraints](https://mlsanity.com/paper/1807.03223/) (2018)
- [Smoother Entropy for Active State Trajectory Estimation and Obfuscation in POMDPs](https://mlsanity.com/paper/2108.10227/) (2021)
- [Probabilistic Opacity for Markov Decision Processes](https://mlsanity.com/paper/1407.4225/) (2014)
- [Quantifying Opacity](https://mlsanity.com/paper/1301.6799/) (2013)
- [Strong current-state and initial-state opacity of discrete-event systems](https://mlsanity.com/paper/2109.05475/) (2021)
- [Synthesis of Opacity-Enforcing Winning Strategies Against Colluded Opponent](https://mlsanity.com/paper/2304.01286/) (2023)
- [Covert Planning against Imperfect Observers](https://mlsanity.com/paper/2310.16791/) (2023)

## Built on by (most cited first, 2 of 2)

- [Synthesis of Dynamic Masks for Information-Theoretic Opacity in Stochastic Systems](https://mlsanity.com/paper/2502.10552/) (2025)
- [Active Perception with Initial-State Uncertainty: A Policy Gradient Method](https://mlsanity.com/paper/2409.16439/) (2024)

## Abstract (arXiv)

The paper studies information-theoretic opacity, an information-flow privacy property, in a setting involving two agents: A planning agent who controls a stochastic system and an observer who partially observes the system states. The goal of the observer is to infer some secret, represented by a random variable, from its partial observations, while the goal of the planning agent is to make the secret maximally opaque to the observer while achieving a satisfactory total return. Modeling the stochastic system using a Markov decision process, two classes of opacity properties are considered -- Last-state opacity is to ensure that the observer is uncertain if the last state is in a specific set and initial-state opacity is to ensure that the observer is unsure of the realization of the initial state. As the measure of opacity, we employ the Shannon conditional entropy capturing the information about the secret revealed by the observable. Then, we develop primal-dual policy gradient methods for opacity-enforcement planning subject to constraints on total returns. We propose novel algorithms to compute the policy gradient of entropy for each observation, leveraging message passing within the hidden Markov models. This gradient computation enables us to have stable and fast convergence. We demonstrate our solution of opacity-enforcement control through a grid world example.