mlsanityData to 7 Oct 2026

Offline Reinforcement Learning with Differential Privacy

arXiv 2206.00810 ↗First posted 2 Jun 2022cs.LG cs.AI cs.CRcited by 19

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The offline reinforcement learning (RL) problem is often motivated by the need to learn data-driven decision policies in financial, legal and healthcare applications. However, the learned policy could retain sensitive information of individuals in the training data (e.g., treatment and outcome of patients), thus susceptible to various privacy risks. We design offline RL algorithms with differential privacy guarantees which provably prevent such risks. These algorithms also enjoy strong instance-dependent learning bounds under both tabular and linear Markov decision process (MDP) settings. Our theory and simulation suggest that the privacy guarantee comes at (almost) no drop in utility comparing to the non-private counterpart for a medium-size dataset.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 50 of its 77 references that are in the map. First posted on arXiv 2 Jun 2022.

Builds on

50 of 77 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism11 Mar
  2. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism17 Oct
  3. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  4. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  5. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  6. 2021Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settings13 May
  7. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  8. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct

Other references · 42

  1. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  2. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  3. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  4. 2016Concentrated Differential Privacy: Simplifications, Extensions, and Lower Boundscited by 339
  5. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  6. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  7. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  8. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 199
  9. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  10. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
  11. 2016Concentrated Differential Privacycited by 139
  12. 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
30 more
  1. 2020When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?cited by 102
  2. 2018Differentially Private Contextual Linear Banditscited by 82
  3. 2019Deep Reinforcement Learning for Tradingcited by 73
  4. 2017Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approachcited by 68
  5. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  6. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  7. 2020Locally Differentially Private (Contextual) Bandits Learningcited by 52
  8. 2020Private Reinforcement Learning with PAC and Regret Guaranteescited by 51
  9. 2017The Price of Differential Privacy For Online Learningcited by 44
  10. 2021A Provably Efficient Algorithm for Linear Markov Decision Process with Low Switching Costcited by 38
  11. 2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
  12. 2022Sample-Efficient Reinforcement Learning with loglog(T) Switching Costcited by 32
  13. 2019Privacy-preserving Q-Learning with Functional Noise in Continuous State Spacescited by 29
  14. 2019Differential Privacy for Multi-armed Bandits: What Is It and What Is Its Cost?cited by 27
  15. 2017Achieving Privacy in the Adversarial Multi-Armed Banditcited by 24
  16. 2021Differentially Private Regret Minimization in Episodic Markov Decision Processescited by 23
  17. 2020Local Differential Privacy for Regret Minimization in Reinforcement Learningcited by 22
  18. 2022Differentially Private Reinforcement Learning with Linear Function Approximationcited by 22
  19. 2016Differentially Private Policy Evaluationcited by 19
  20. 2020(Locally) Differentially Private Combinatorial Semi-Banditscited by 14
  21. 2021Locally Differentially Private Reinforcement Learning for Linear Mixture Markov Decision Processescited by 13
  22. 2021Privately Publishable Per-instance Privacycited by 12
  23. 2021Differentially Private Exploration in Reinforcement Learning with Linear Representationcited by 11
  24. 2022Differentially Private Linear Sketches: Efficient Implementations and Applicationscited by 10
  25. 2020Locally Private Distributed Reinforcement Learningcited by 9
  26. 2022Improved Regret for Differentially Private Exploration in Linear MDPcited by 8
  27. 2021Adaptive Control of Differentially Private Linear Quadratic Systemscited by 7
  28. 2020Privacy-Constrained Policies via Mutual Information Regularized Policy Gradientscited by 6
  29. 2019Actor Critic with Differentially Private Criticcited by 5
  30. 2019Privacy Preserving Off-Policy Evaluationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 9; 2025: 3; 2026: 2.

19 papers.

Sort:

By area:

  1. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
  2. 2023Privately Aligning Language Models with Reinforcement Learningcited by 17
  3. 2022Near-Optimal Differentially Private Reinforcement Learningcited by 16
  4. 2024Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescited by 12
  5. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 12
  6. 2022Doubly Fair Dynamic Pricingcited by 8
  7. 2024Thinking Racial Bias in Fair Forgery Detection: Models, Datasets and Evaluationscited by 8
  8. 2025Privacy-R1: Privacy-Aware Multi-LLM Agent Collaboration via Reinforcement Learningcited by 6
  9. 2023Differential Privacy in Cooperative Multiagent Planningcited by 2
  10. 2024Offline Behavior Distillationcited by 2
  11. 2024Differentially Private Deep Model-Based Reinforcement Learningcited by 1
  12. 2024Differentially Private Reinforcement Learning with Self-Playcited by 1
7 more
  1. 2024On the Exponential Convergence for Offline RLHF with Pairwise Comparisonscited by 1
  2. 2025State Diversity Matters in Offline Behavior Distillationcited by 1
  3. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Datacited by 0
  4. 2024Preserving Expert-Level Privacy in Offline Reinforcement Learningcited by 0
  5. 2025Towards Optimal Differentially Private Regret Bounds in Linear MDPscited by 0
  6. 2026Non-Parametric Rehearsal Learning via Conditional Mean Embeddingscited by 0
  7. 2026Privacy Preserving Reinforcement Learning with One-Sided Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling
  2. 2025PrivORL: Differentially Private Synthetic Dataset for Offline Reinforcement Learning
  3. 2025Towards User-level Private Reinforcement Learning with Human Feedback
  4. 2025Position Paper: Rethinking Privacy in RL for Sequential Decision-making in the Age of LLMs
  5. 2025Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learning
  6. 2025KL-regularization Itself is Differentially Private in Bandits and RLHF
  7. 2023The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learning
  8. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning