mlsanityData to 14 Oct 2026

Differentially Private Reinforcement Learning with Linear Function Approximation

arXiv 2201.07052 ↗First posted 18 Jan 2022cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Motivated by the wide adoption of reinforcement learning (RL) in real-world personalized services, where users' sensitive and private information needs to be protected, we study regret minimization in finite-horizon Markov decision processes (MDPs) under the constraints of differential privacy (DP). Compared to existing private RL algorithms that work only on tabular finite-state, finite-actions MDPs, we take the first step towards privacy-preserving learning in MDPs with large state and action spaces. Specifically, we consider MDPs with linear function approximation (in particular linear mixture MDPs) under the notion of joint differential privacy (JDP), where the RL agent is responsible for protecting users' sensitive data. We design two private RL algorithms that are based on value iteration and policy optimization, respectively, and show that they enjoy sub-linear regret performance while guaranteeing privacy protection. Moreover, the regret bounds are independent of the number of states, and scale at most logarithmically with the number of actions, making the algorithms suitable for privacy protection in nowadays large-scale personalized services. Our results are achieved via a general procedure for learning in linear mixture MDPs under changing regularizers, which not only generalizes previous results for non-private learning, but also serves as a building block for general private reinforcement learning.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 35 of its 66 references that are in the map. First posted on arXiv 18 Jan 2022.

Builds on

35 of 66 references.

References · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  4. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  5. 2016Concentrated Differential Privacy: Simplifications, Extensions, and Lower Boundscited by 346
  6. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 234
  7. 2019Provably Efficient Exploration in Policy Optimizationcited by 205
  8. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 200
  9. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
  10. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  11. 2019Learning with Good Feature Representations in Bandits and in RL with a Generative Modelcited by 127
  12. 2018Distributed Differential Privacy via Shufflingcited by 115
23 more
  1. 2020Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mappingcited by 102
  2. 2018Differentially Private Contextual Linear Banditscited by 82
  3. 2020Optimistic Policy Optimization with Bandit Feedbackcited by 79
  4. 2020Logarithmic Regret for Reinforcement Learning with Linear Function Approximationcited by 79
  5. 2020Locally Differentially Private (Contextual) Bandits Learningcited by 53
  6. 2020Private Reinforcement Learning with PAC and Regret Guaranteescited by 51
  7. 2015Algorithms for Differentially Private Multi-Armed Banditscited by 45
  8. 2017The Price of Differential Privacy For Online Learningcited by 45
  9. 2019Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policycited by 43
  10. 2019An Optimal Private Stochastic-MAB Algorithm Based on an Optimal Private Stopping Rulecited by 37
  11. 2020Multi-Armed Bandits with Local Differential Privacycited by 37
  12. 2019Privacy-preserving Q-Learning with Functional Noise in Continuous State Spacescited by 30
  13. 2019Differential Privacy for Multi-armed Bandits: What Is It and What Is Its Cost?cited by 27
  14. 2017Achieving Privacy in the Adversarial Multi-Armed Banditcited by 24
  15. 2013Private Matchings and Allocationscited by 23
  16. 2021Differentially Private Regret Minimization in Episodic Markov Decision Processescited by 23
  17. 2016Differentially Private Policy Evaluationcited by 19
  18. 2020Local Differential Privacy for Bayesian Optimizationcited by 19
  19. 2021Locally Differentially Private Reinforcement Learning for Linear Mixture Markov Decision Processescited by 13
  20. 2021No-Regret Algorithms for Private Gaussian Process Bandit Optimizationcited by 11
  21. 2021Differentially Private Exploration in Reinforcement Learning with Linear Representationcited by 11
  22. 2020Locally Private Distributed Reinforcement Learningcited by 9
  23. 2021Adaptive Control of Differentially Private Linear Quadratic Systemscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 5; 2024: 6; 2025: 3; 2026: 2.

22 papers.

Sort:

By area:

  1. 2022Shuffle Private Linear Contextual Banditscited by 27
  2. 2022Offline Reinforcement Learning with Differential Privacycited by 20
  3. 2022Near-Optimal Differentially Private Reinforcement Learningcited by 16
  4. 2022Distributed Differential Privacy in Multi-Armed Banditscited by 14
  5. 2023On Differentially Private Federated Linear Contextual Banditscited by 14
  6. 2022Improved Regret for Differentially Private Exploration in Linear MDPcited by 8
  7. 2022Differentially Private Temporal Difference Learning with Stochastic Nonconvex-Strongly-Concave Optimizationcited by 7
  8. 2023Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challengescited by 5
  9. 2025Improved Bounds for Private and Robust Alignmentcited by 5
  10. 2023Differentially Private Episodic Reinforcement Learning with Heavy-tailed Rewardscited by 4
  11. 2025SquarechiPO: Differentially Private and Robust chi2-Preference Optimization in Offline Direct Alignmentcited by 3
  12. 2025On the Sample Complexity of Differentially Private Policy Optimizationcited by 3
10 more
  1. 2023DPMAC: Differentially Private Communication for Cooperative Multi-Agent Reinforcement Learningcited by 1
  2. 2023Privacy-Engineered Value Decomposition Networks for Cooperative Multi-Agent Reinforcement Learningcited by 1
  3. 2024Differentially Private Deep Model-Based Reinforcement Learningcited by 1
  4. 2024Differentially Private Reinforcement Learning with Self-Playcited by 1
  5. 2024Privacy Preserving Reinforcement Learning for Population Processescited by 1
  6. 2024Preserving the Privacy of Reward Functions in MDPs through Deceptioncited by 1
  7. 2024Near-Optimal Reinforcement Learning with Shuffle Differential Privacycited by 1
  8. 2026Exact Unlearning in Reinforcement Learningcited by 1
  9. 2024Preserving Expert-Level Privacy in Offline Reinforcement Learningcited by 0
  10. 2026ForgetMimic: Motion Unlearning for Reinforcement Learning Humanoid Controlcited by 0