# Policy Gradient Methods for Information-Theoretic Opacity in Markov Decision Processes

arXiv: https://arxiv.org/abs/2511.02704 · HTML: https://mlsanity.com/paper/2511.02704/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 39 references that are in the map.
- First posted on arXiv 4 Nov 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 14 of 14)

- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Bridging the Gap Between Value and Policy Based Reinforcement Learning](https://mlsanity.com/paper/1702.08892/) (2017)
- [Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization](https://mlsanity.com/paper/2007.06558/) (2020)
- [On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning](https://mlsanity.com/paper/1704.00805/) (2017)
- [Entropy Maximization for Markov Decision Processes Under Temporal Logic Constraints](https://mlsanity.com/paper/1807.03223/) (2018)
- [Smoother Entropy for Active State Trajectory Estimation and Obfuscation in POMDPs](https://mlsanity.com/paper/2108.10227/) (2021)
- [Differential Privacy for Symbolic Systems with Application to Markov Chains](https://mlsanity.com/paper/2202.03325/) (2022)
- [Probabilistic Opacity for Markov Decision Processes](https://mlsanity.com/paper/1407.4225/) (2014)
- [Quantifying Opacity](https://mlsanity.com/paper/1301.6799/) (2013)
- [Strong current-state and initial-state opacity of discrete-event systems](https://mlsanity.com/paper/2109.05475/) (2021)
- [Synthesis of Opacity-Enforcing Winning Strategies Against Colluded Opponent](https://mlsanity.com/paper/2304.01286/) (2023)
- [Active Inference through Incentive Design in Markov Decision Processes](https://mlsanity.com/paper/2502.07065/) (2025)
- [On Approximate Opacity of Cyber-Physical Systems](https://mlsanity.com/paper/1902.09411/) (2019)
- [Synthesis of Dynamic Masks for Information-Theoretic Opacity in Stochastic Systems](https://mlsanity.com/paper/2502.10552/) (2025)

## Abstract (arXiv)

Opacity, or non-interference, is a property ensuring that an external observer cannot infer confidential information (the "secret") from system observations. We introduce an information-theoretic measure of opacity, which quantifies information leakage using the conditional entropy of the secret given the observer's partial observations in a system modeled as a Markov decision process (MDP). Our objective is to find a control policy that maximizes opacity while satisfying task performance constraints, assuming that an informed observer is aware of the control policy and system dynamics. Specifically, we consider a class of opacity called state-based opacity, where the secret is a propositional formula about the past or current state of the system, and a special case of state-based opacity called language-based opacity, where the secret is defined by a temporal logic formula (LTL) or a regular language recognized by a finite-state automaton. First, we prove that finite-memory policies can outperform Markov policies in optimizing information-theoretic opacity. Second, we develop an algorithm to compute a maximally opaque Markov policy using a primal-dual gradient-based algorithm, and prove its convergence. Since opacity cannot be expressed as a cumulative cost, we develop a novel method to compute the gradient of conditional entropy with respect to policy parameters using observable operators in hidden Markov models. The experimental results validate the effectiveness and optimality of our proposed methods.