mlsanityData to 14 Oct 2026

Preventing Imitation Learning with Adversarial Policy Ensembles

arXiv 2002.01059 ↗First posted 31 Jan 2020cs.LG cs.AI stat.MLcited by 0

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Imitation learning can reproduce policies by observing experts, which poses a problem regarding policy privacy. Policies, such as human, or policies on deployed robots, can all be cloned without consent from the owners. How can we protect against external observers cloning our proprietary policies? To answer this question we introduce a new reinforcement learning framework, where we train an ensemble of near-optimal policies, whose demonstrations are guaranteed to be useless for an external observer. We formulate this idea by a constrained optimization problem, where the objective is to improve proprietary policies, and at the same time deteriorate the virtual policy of an eventual external observer. We design a tractable algorithm to solve this new optimization problem by modifying the standard policy gradient algorithm. Our formulation can be interpreted in lenses of confidentiality and adversarial behaviour, which enables a broader perspective of this work. We demonstrate the existence of "non-clonable" ensembles, providing a solution to the above optimization problem, which is calculated by our modified policy gradient algorithm. To our knowledge, this is the first work regarding the protection of policies in Reinforcement Learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 42 references that are in the map. First posted on arXiv 31 Jan 2020.

Builds on

31 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2019Causal Confusion in Imitation Learning28 May
  2. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow1 Oct

Other references · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2016Generative Adversarial Imitation Learningcited by 1,597
  7. 2016End to End Learning for Self-Driving Carscited by 1,009
  8. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  9. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  10. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  11. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 348
  12. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 346
17 more
  1. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  2. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 228
  3. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 214
  4. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 166
  5. 2018Variational Option Discovery Algorithmscited by 122
  6. 2017Vulnerability of Deep Reinforcement Learning to Policy Induction Attackscited by 91
  7. 2018Privacy Preserving Machine Learning: Threats and Solutionscited by 79
  8. 2019InfoBot: Transfer and Exploration via the Information Bottleneckcited by 78
  9. 2019Policy Poisoning in Batch Reinforcement Learning and Controlcited by 70
  10. 2017OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learningcited by 40
  11. 2019Privacy-preserving Q-Learning with Functional Noise in Continuous State Spacescited by 30
  12. 2019Risk-Aware Active Inverse Reinforcement Learningcited by 26
  13. 2019Learning Novel Policies For Taskscited by 20
  14. 2016Differentially Private Policy Evaluationcited by 19
  15. 2018Learning to Share and Hide Intentions using Information Regularizationcited by 13
  16. 2017InfoGAIL: Interpretable Imitation Learning from Visual Demonstrationscited by 11
  17. 2018Interpretable Latent Spaces for Learning from Demonstrationcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
  2. 2022Imitation Learning by State-Only Distribution Matching
  3. 2020Adversarial Imitation Learning via Random Search
  4. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
  5. 2021Robust Maximum Entropy Behavior Cloning
  6. 2021Imitation Learning by Reinforcement Learning
  7. 2022Planning for Sample Efficient Imitation Learning
  8. 2024Efficient Imitation Learning with Conservative World Models