Divide and Repair: Using Options to Improve Performance of Imitation Learning Against Adversarial Demonstrations
On the map: Offline RL and imitation learning from data → Imitation learning methods
Abstract (arXiv)
We consider the problem of learning to perform a task from demonstrations given by teachers or experts, when some of the experts' demonstrations might be adversarial and demonstrate an incorrect way to perform the task. We propose a novel technique that can identify parts of demonstrated trajectories that have not been significantly modified by the adversary and utilize them for learning, using temporally extended policies or options. We first define a trajectory divergence measure based on the spatial and temporal features of demonstrated trajectories to detect and discard parts of the trajectories that have been significantly modified by an adversarial expert, and, could degrade the learner's performance, if used for learning, We then use an options-based algorithm that partitions trajectories and learns only from the parts of trajectories that have been determined as admissible. We provide theoretical results of our technique to show that repairing partial trajectories improves the sample efficiency of the demonstrations without degrading the learner's performance. We then evaluate the proposed algorithm for learning to play an Atari-like, computer-based game called LunarLander in the presence of different types and degrees of adversarial attacks of demonstrated trajectories. Our experimental results show that our technique can identify adversarially modified parts of the demonstrated trajectories and successfully prevent the learning performance from degrading due to adversarial demonstrations.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 41 references that are in the map. First posted on arXiv 7 Jun 2023.
Builds on
17 of 41 references.
Closest ancestors · 4
Same direction, newest first
- 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamics10 Mar
- 2019Recent Advances in Imitation Learning from Observation30 May
- 2019Adversarial Imitation Learning from Incomplete Demonstrations29 May
- 2019Imitation Learning from Imperfect Demonstration27 Jan
Other references · 13
- 2014Explaining and Harnessing Adversarial Examplescited by 7,287
- 2022Generative Adversarial Networkscited by 5,603
- 2016Generative Adversarial Imitation Learningcited by 1,576
- 2017Robust Adversarial Reinforcement Learningcited by 439
- 2016The Option-Critic Architecturecited by 433
- 2017Adversarial Attacks on Neural Network Policiescited by 314
- 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
- 2018Adversarial Attacks and Defences: A Surveycited by 259
- 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
- 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 209
- 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 163
- 2019Risk Averse Robust Adversarial Reinforcement Learningcited by 54
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Unlabeled Imperfect Demonstrations in Adversarial Imitation Learning
- 2019Sample-efficient Adversarial Imitation Learning from Observation
- 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations
- 2022Learning from Imperfect Demonstrations via Adversarial Confidence Transfer
- 2022Understanding Adversarial Imitation Learning in Small Sample Regime: A Stage-coupled Analysis
- 2020ADAIL: Adaptive Adversarial Imitation Learning
- 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitions
- 2021What Matters for Adversarial Imitation Learning?