# Privacy-preserving Q-Learning with Functional Noise in Continuous State Spaces

arXiv: https://arxiv.org/abs/1901.10634 · HTML: https://mlsanity.com/paper/1901.10634/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2019.

## Built on by (most cited first, 30 of 30)

- [Differentially Private Regret Minimization in Episodic Markov Decision Processes](https://mlsanity.com/paper/2112.10599/) (2021)
- [Differentially Private Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/2201.07052/) (2022)
- [Offline Reinforcement Learning with Differential Privacy](https://mlsanity.com/paper/2206.00810/) (2022)
- [Privately Aligning Language Models with Reinforcement Learning](https://mlsanity.com/paper/2310.16960/) (2023)
- [Deceptive Reinforcement Learning for Privacy-Preserving Planning](https://mlsanity.com/paper/2102.03022/) (2021)
- [A Review of Safe Reinforcement Learning Methods for Modern Power Systems](https://mlsanity.com/paper/2407.00304/) (2024)
- [Towards Quantifying the Carbon Emissions of Differentially Private Machine Learning](https://mlsanity.com/paper/2107.06946/) (2021)
- [Differentially Private Temporal Difference Learning with Stochastic Nonconvex-Strongly-Concave Optimization](https://mlsanity.com/paper/2201.10447/) (2022)
- [New Challenges in Reinforcement Learning: A Survey of Security and Privacy](https://mlsanity.com/paper/2301.00188/) (2022)
- [adaPARL: Adaptive Privacy-Aware Reinforcement Learning for Sequential-Decision Making Human-in-the-Loop Systems](https://mlsanity.com/paper/2303.04257/) (2023)
- [How Private Is Your RL Policy? An Inverse RL Based Analysis Framework](https://mlsanity.com/paper/2112.05495/) (2021)
- [Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges](https://mlsanity.com/paper/2305.10091/) (2023)
- [Privacy of Autonomous Vehicles: Risks, Protection Methods, and Future Directions](https://mlsanity.com/paper/2209.04022/) (2022)
- [Privacy Risks in Reinforcement Learning for Household Robots](https://mlsanity.com/paper/2306.09273/) (2023)
- [Hiding in Plain Sight: Differential Privacy Noise Exploitation for Evasion-resilient Localized Poisoning Attacks in Multiagent Reinforcement Learning](https://mlsanity.com/paper/2307.00268/) (2023)
- [Improved Algorithms for Differentially Private Language Model Alignment](https://mlsanity.com/paper/2505.08849/) (2025)
- [Reinforcement Learning on Encrypted Data](https://mlsanity.com/paper/2109.08236/) (2021)
- [Provably Efficient Convergence of Primal-Dual Actor-Critic with Nonlinear Function Approximation](https://mlsanity.com/paper/2202.13863/) (2022)
- [Privacy-Preserving Reinforcement Learning Beyond Expectation](https://mlsanity.com/paper/2203.10165/) (2022)
- [BRNES: Enabling Security and Privacy-aware Experience Sharing in Multiagent Robotic and Autonomous Systems](https://mlsanity.com/paper/2308.01274/) (2023)
- [DPMAC: Differentially Private Communication for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2308.09902/) (2023)
- [Privacy-Engineered Value Decomposition Networks for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2311.06255/) (2023)
- [Differentially Private Deep Model-Based Reinforcement Learning](https://mlsanity.com/paper/2402.05525/) (2024)
- [PEARL: Structural Privacy-Utility Control in Human-Centric CPS via Personalized Early-Exit Deep Reinforcement Learning](https://mlsanity.com/paper/2403.05864/) (2024)
- [Differentially Private Reinforcement Learning with Self-Play](https://mlsanity.com/paper/2404.07559/) (2024)
- [Privacy Preserving Reinforcement Learning for Population Processes](https://mlsanity.com/paper/2406.17649/) (2024)
- [Preserving the Privacy of Reward Functions in MDPs through Deception](https://mlsanity.com/paper/2407.09809/) (2024)
- [Preventing Imitation Learning with Adversarial Policy Ensembles](https://mlsanity.com/paper/2002.01059/) (2020)
- [Differentially Private Deep Q-Learning for Pattern Privacy Preservation in MEC Offloading](https://mlsanity.com/paper/2302.04608/) (2023)
- [Preserving Expert-Level Privacy in Offline Reinforcement Learning](https://mlsanity.com/paper/2411.13598/) (2024)

## Abstract (arXiv)

We consider differentially private algorithms for reinforcement learning in continuous spaces, such that neighboring reward functions are indistinguishable. This protects the reward information from being exploited by methods such as inverse reinforcement learning. Existing studies that guarantee differential privacy are not extendable to infinite state spaces, as the noise level to ensure privacy will scale accordingly to infinity. Our aim is to protect the value function approximator, without regard to the number of states queried to the function. It is achieved by adding functional noise to the value function iteratively in the training. We show rigorous privacy guarantees by a series of analyses on the kernel of the noise space, the probabilistic bound of such noise samples, and the composition over the iterations. We gain insight into the utility analysis by proving the algorithm's approximate optimality when the state space is discrete. Experiments corroborate our theoretical findings and show improvement over existing approaches.