mlsanityData to 7 Oct 2026

Differentially Private Deep Model-Based Reinforcement Learning

arXiv 2402.05525 ↗First posted 8 Feb 2024cs.LG cs.AI cs.CRcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We address private deep offline reinforcement learning (RL), where the goal is to train a policy on standard control tasks that is differentially private (DP) with respect to individual trajectories in the dataset. To achieve this, we introduce PriMORL, a model-based RL algorithm with formal differential privacy guarantees. PriMORL first learns an ensemble of trajectory-level DP models of the environment from offline data. It then optimizes a policy on the penalized private model, without any further interaction with the system or access to the dataset. In addition to offering strong theoretical foundations, we demonstrate empirically that PriMORL enables the training of private RL agents on offline continuous control tasks with deep function approximations, whereas current methods are limited to simpler tabular and linear Markov Decision Processes (MDPs). We furthermore outline the trade-offs involved in achieving privacy in this setting.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 41 of its 56 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

41 of 56 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Model-based Offline Reinforcement Learning with Count-based Conservatism21 Jul
  2. 2022Offline Reinforcement Learning with Differential Privacy2 Jun
  3. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  4. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  5. 2020MOPO: Model-based Offline Policy Optimization27 May
  6. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 33

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2016Deep Learning with Differential Privacycited by 2,682
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2018DeepMind Control Suitecited by 935
  8. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  10. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  11. 2017Renyi Differential Privacycited by 643
  12. 2016Concentrated Differential Privacy: Simplifications, Extensions, and Lower Boundscited by 339
21 more
  1. 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 237
  2. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  3. 2020A Survey of Privacy Attacks in Machine Learningcited by 143
  4. 2016Concentrated Differential Privacycited by 139
  5. 2020Model-Based Offline Planningcited by 126
  6. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  7. 2020Private Reinforcement Learning with PAC and Regret Guaranteescited by 51
  8. 2015Algorithms for Differentially Private Multi-Armed Banditscited by 44
  9. 2019Privacy-preserving Q-Learning with Functional Noise in Continuous State Spacescited by 29
  10. 2019How You Act Tells a Lot: Privacy-Leakage Attack on Deep Reinforcement Learningcited by 29
  11. 2019Differential Privacy for Multi-armed Bandits: What Is It and What Is Its Cost?cited by 27
  12. 2021Differentially Private Regret Minimization in Episodic Markov Decision Processescited by 23
  13. 2020Local Differential Privacy for Regret Minimization in Reinforcement Learningcited by 22
  14. 2022Differentially Private Reinforcement Learning with Linear Function Approximationcited by 22
  15. 2022Near-Optimal Differentially Private Reinforcement Learningcited by 16
  16. 2021Locally Differentially Private Reinforcement Learning for Linear Mixture Markov Decision Processescited by 13
  17. 2021Differentially Private Exploration in Reinforcement Learning with Linear Representationcited by 11
  18. 2022Improved Regret for Differentially Private Exploration in Linear MDPcited by 8
  19. 2020Privacy-Constrained Policies via Mutual Information Regularized Policy Gradientscited by 6
  20. 2021How Private Is Your RL Policy? An Inverse RL Based Analysis Frameworkcited by 5
  21. 2020Data Heterogeneity Differential Privacy: From Theory to Algorithmcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Decoupling Identity from Utility: Privacy-by-Design Frameworks for Financial Ecosystemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PrivORL: Differentially Private Synthetic Dataset for Offline Reinforcement Learning
  2. 2024Preserving Expert-Level Privacy in Offline Reinforcement Learning
  3. 2026Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling
  4. 2026Reward-Driven Learning under Prompt-Level Differential Privacy
  5. 2025KL-regularization Itself is Differentially Private in Bandits and RLHF
  6. 2025Position Paper: Rethinking Privacy in RL for Sequential Decision-making in the Age of LLMs
  7. 2025Towards User-level Private Reinforcement Learning with Human Feedback
  8. 2025Privacy-R1: Privacy-Aware Multi-LLM Agent Collaboration via Reinforcement Learning