# How Private Is Your RL Policy? An Inverse RL Based Analysis Framework

arXiv: https://arxiv.org/abs/2112.05495 · HTML: https://mlsanity.com/paper/2112.05495/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 10 Dec 2021.

## Builds on (most cited first, 17 of 17)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Adversarial Attacks on Neural Network Policies](https://mlsanity.com/paper/1702.02284/) (2017)
- [Adversarial Policies: Attacking Deep Reinforcement Learning](https://mlsanity.com/paper/1905.10615/) (2019)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Delving into adversarial attacks on deep policies](https://mlsanity.com/paper/1705.06452/) (2017)
- [Robust Deep Reinforcement Learning through Adversarial Loss](https://mlsanity.com/paper/2008.01976/) (2020)
- [Private Reinforcement Learning with PAC and Regret Guarantees](https://mlsanity.com/paper/2009.09052/) (2020)
- [Privacy-preserving Q-Learning with Functional Noise in Continuous State Spaces](https://mlsanity.com/paper/1901.10634/) (2019)
- [How You Act Tells a Lot: Privacy-Leakage Attack on Deep Reinforcement Learning](https://mlsanity.com/paper/1904.11082/) (2019)
- [Differentially Private Policy Evaluation](https://mlsanity.com/paper/1603.02010/) (2016)
- [Actor Critic with Differentially Private Critic](https://mlsanity.com/paper/1910.05876/) (2019)
- [Privacy-Preserving Multi-Party Contextual Bandits](https://mlsanity.com/paper/1910.05299/) (2019)

## Built on by (most cited first, 5 of 5)

- [New Challenges in Reinforcement Learning: A Survey of Security and Privacy](https://mlsanity.com/paper/2301.00188/) (2022)
- [Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges](https://mlsanity.com/paper/2305.10091/) (2023)
- [BRNES: Enabling Security and Privacy-aware Experience Sharing in Multiagent Robotic and Autonomous Systems](https://mlsanity.com/paper/2308.01274/) (2023)
- [Differentially Private Deep Model-Based Reinforcement Learning](https://mlsanity.com/paper/2402.05525/) (2024)
- [Preserving the Privacy of Reward Functions in MDPs through Deception](https://mlsanity.com/paper/2407.09809/) (2024)

## Abstract (arXiv)

Reinforcement Learning (RL) enables agents to learn how to perform various tasks from scratch. In domains like autonomous driving, recommendation systems, and more, optimal RL policies learned could cause a privacy breach if the policies memorize any part of the private reward. We study the set of existing differentially-private RL policies derived from various RL algorithms such as Value Iteration, Deep Q Networks, and Vanilla Proximal Policy Optimization. We propose a new Privacy-Aware Inverse RL (PRIL) analysis framework, that performs reward reconstruction as an adversarial attack on private policies that the agents may deploy. For this, we introduce the reward reconstruction attack, wherein we seek to reconstruct the original reward from a privacy-preserving policy using an Inverse RL algorithm. An adversary must do poorly at reconstructing the original reward function if the agent uses a tightly private policy. Using this framework, we empirically test the effectiveness of the privacy guarantee offered by the private algorithms on multiple instances of the FrozenLake domain of varying complexities. Based on the analysis performed, we infer a gap between the current standard of privacy offered and the standard of privacy needed to protect reward functions in RL. We do so by quantifying the extent to which each private policy protects the reward function by measuring distances between the original and reconstructed rewards.