# Privacy-Preserving Reinforcement Learning Beyond Expectation

arXiv: https://arxiv.org/abs/2203.10165 · HTML: https://mlsanity.com/paper/2203.10165/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 54 references that are in the map.
- First posted on arXiv 18 Mar 2022.

## Builds on (most cited first, 17 of 17)

- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Risk-sensitive Reinforcement Learning](https://mlsanity.com/paper/1311.2097/) (2013)
- [Deep Learning in Mobile and Wireless Networking: A Survey](https://mlsanity.com/paper/1803.04311/) (2018)
- [How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Robotics](https://mlsanity.com/paper/1710.11040/) (2017)
- [Private Reinforcement Learning with PAC and Regret Guarantees](https://mlsanity.com/paper/2009.09052/) (2020)
- [Privacy-preserving Q-Learning with Functional Noise in Continuous State Spaces](https://mlsanity.com/paper/1901.10634/) (2019)
- [Local Differential Privacy for Regret Minimization in Reinforcement Learning](https://mlsanity.com/paper/2010.07778/) (2020)
- [Differentially Private Policy Evaluation](https://mlsanity.com/paper/1603.02010/) (2016)
- [FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback](https://mlsanity.com/paper/2001.06781/) (2020)
- [Constrained Risk-Averse Markov Decision Processes](https://mlsanity.com/paper/2012.02423/) (2020)
- [Risk-sensitive safety analysis using Conditional Value-at-Risk](https://mlsanity.com/paper/2101.12086/) (2021)
- [Privacy-Preserving Policy Synthesis in Markov Decision Processes](https://mlsanity.com/paper/2004.07778/) (2020)
- [Prospect-theoretic Q-learning](https://mlsanity.com/paper/2104.05311/) (2021)
- [Differentially Private LQ Control](https://mlsanity.com/paper/1807.05082/) (2018)
- [Reinforcement Learning Beyond Expectation](https://mlsanity.com/paper/2104.00540/) (2021)
- [Prospect Theoretic Analysis of Privacy-Preserving Mechanism](https://mlsanity.com/paper/1911.03833/) (2019)
- [Privacy-Preserving Resilience of Cyber-Physical Systems to Adversaries](https://mlsanity.com/paper/2007.13272/) (2020)

## Built on by (most cited first, 1 of 1)

- [Near-Optimal Reinforcement Learning with Shuffle Differential Privacy](https://mlsanity.com/paper/2411.11647/) (2024)

## Abstract (arXiv)

Cyber and cyber-physical systems equipped with machine learning algorithms such as autonomous cars share environments with humans. In such a setting, it is important to align system (or agent) behaviors with the preferences of one or more human users. We consider the case when an agent has to learn behaviors in an unknown environment. Our goal is to capture two defining characteristics of humans: i) a tendency to assess and quantify risk, and ii) a desire to keep decision making hidden from external parties. We incorporate cumulative prospect theory (CPT) into the objective of a reinforcement learning (RL) problem for the former. For the latter, we use differential privacy. We design an algorithm to enable an RL agent to learn policies to maximize a CPT-based objective in a privacy-preserving manner and establish guarantees on the privacy of value functions learned by the algorithm when rewards are sufficiently close. This is accomplished through adding a calibrated noise using a Gaussian process mechanism at each step. Through empirical evaluations, we highlight a privacy-utility tradeoff and demonstrate that the RL agent is able to learn behaviors that are aligned with that of a human user in the same environment in a privacy-preserving manner