mlsanityData to 14 Oct 2026

Preserving Expert-Level Privacy in Offline Reinforcement Learning

arXiv 2411.13598 ↗First posted 18 Nov 2024cs.CR cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The offline reinforcement learning (RL) problem aims to learn an optimal policy from historical data collected by one or more behavioural policies (experts) by interacting with an environment. However, the individual experts may be privacy-sensitive in that the learnt policy may retain information about their precise choices. In some domains like personalized retrieval, advertising and healthcare, the expert choices are considered sensitive data. To provably protect the privacy of such experts, we propose a novel consensus-based expert-level differentially private offline RL training approach compatible with any existing offline RL algorithm. We prove rigorous differential privacy guarantees, while maintaining strong empirical performance. Unlike existing work in differentially private RL, we supplement the theory with proof-of-concept experiments on classic RL environments featuring large continuous state spaces, demonstrating substantial improvements over a natural baseline across multiple tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 52 references that are in the map. First posted on arXiv 18 Nov 2024.

Builds on

38 of 52 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare2 May
  2. 2022Offline Reinforcement Learning with Differential Privacy2 Jun
  3. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  4. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  5. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020MOPO: Model-based Offline Policy Optimization27 May
  9. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  12. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 26

  1. 2016Deep Learning with Differential Privacycited by 2,728
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  4. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 348
  5. 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 242
  6. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  7. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 170
  8. 2018Amplification by Shuffling: From Local to Central Differential Privacy via Anonymitycited by 164
  9. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
  10. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 117
  11. 2020Hiding Among the Clones: A Simple and Nearly Optimal Analysis of Privacy Amplification by Shufflingcited by 96
  12. 2017Real-Time Bidding by Reinforcement Learning in Display Advertisingcited by 91
14 more
  1. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  2. 2021Learning with User-Level Privacycited by 72
  3. 2020Private Reinforcement Learning with PAC and Regret Guaranteescited by 51
  4. 2023Learning to Warm-Start Fixed-Point Optimization Algorithmscited by 32
  5. 2019Privacy-preserving Q-Learning with Functional Noise in Continuous State Spacescited by 30
  6. 2024Fine-Tuning Large Language Models with User-Level Differential Privacycited by 29
  7. 2019Reinforcement Learning for Slate-based Recommender Systems: A Tractable Decomposition and Practical Methodologycited by 27
  8. 2021Differentially Private Regret Minimization in Episodic Markov Decision Processescited by 23
  9. 2022Differentially Private Reinforcement Learning with Linear Function Approximationcited by 22
  10. 2022Near-Optimal Differentially Private Reinforcement Learningcited by 16
  11. 2021Locally Differentially Private Reinforcement Learning for Linear Mixture Markov Decision Processescited by 13
  12. 2023User-level Differentially Private Stochastic Convex Optimization: Efficient Algorithms with Optimal Ratescited by 12
  13. 2024A Huber Loss Minimization Approach to Mean Estimation under User-level Differential Privacycited by 11
  14. 2021Locally Differentially Private Sparse Vector Aggregationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling
  2. 2024Differentially Private Deep Model-Based Reinforcement Learning
  3. 2025Differentially Private Policy Gradient
  4. 2024Differentially Private Reinforcement Learning with Self-Play
  5. 2024Near-Optimal Reinforcement Learning with Shuffle Differential Privacy
  6. 2025PrivORL: Differentially Private Synthetic Dataset for Offline Reinforcement Learning
  7. 2025Towards User-level Private Reinforcement Learning with Human Feedback
  8. 2024Privacy Preserving Reinforcement Learning for Population Processes