Adaptive Discounting of Training Time Attacks
On the map: LLM safety and security → Prompt injection and agent security
Abstract (arXiv)
Among the most insidious attacks on Reinforcement Learning (RL) solutions are training-time attacks (TTAs) that create loopholes and backdoors in the learned behaviour. Not limited to a simple disruption, constructive TTAs (C-TTAs) are now available, where the attacker forces a specific, target behaviour upon a training RL agent (victim). However, even state-of-the-art C-TTAs focus on target behaviours that could be naturally adopted by the victim if not for a particular feature of the environment dynamics, which C-TTAs exploit. In this work, we show that a C-TTA is possible even when the target behaviour is un-adoptable due to both environment dynamics as well as non-optimality with respect to the victim objective(s). To find efficient attacks in this context, we develop a specialised flavour of the DDPG algorithm, which we term gammaDDPG, that learns this stronger version of C-TTA. gammaDDPG dynamically alters the attack policy planning horizon based on the victim's current behaviour. This improves effort distribution throughout the attack timeline and reduces the effect of uncertainty the attacker has about the victim. To demonstrate the features of our method and better relate the results to prior research, we borrow a 3D grid domain from a state-of-the-art C-TTA for our experiments. Code is available at "bit.ly/github-rb-gDDPG".
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 27 of its 52 references that are in the map. First posted on arXiv 5 Jan 2024.
Builds on
27 of 52 references.
References · 27
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
- 2017Adversarial Attacks on Neural Network Policiescited by 314
- 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 229
- 2020Prioritized Level Replaycited by 190
- 2022Evolving Curricula with Regret-Based Environment Designcited by 166
- 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 133
- 2021Replay-Guided Adversarial Environment Designcited by 108
- 2017Delving into adversarial attacks on deep policiescited by 104
- 2021Outside the Echo Chamber: Optimizing the Performative Riskcited by 95
- 2020Stochastic Optimization for Performative Predictioncited by 92
- 2017Vulnerability of Deep Reinforcement Learning to Policy Induction Attackscited by 91
15 more
- 2020Performative Prediction in a Stateful Worldcited by 90
- 2020Adaptive Reward-Poisoning Attacks against Reinforcement Learningcited by 86
- 2020Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learningcited by 81
- 2020Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learningcited by 62
- 2019Certified Adversarial Robustness for Deep Reinforcement Learningcited by 47
- 2022Multiplayer Performative Prediction: Learning in Decision-Dependent Gamescited by 42
- 2016Unifying task specification in reinforcement learningcited by 41
- 2020Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamicscited by 38
- 2015How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategiescited by 26
- 2021Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environmentscited by 26
- 2022Performative Reinforcement Learningcited by 25
- 2021Defense Against Reward Poisoning Attacks in Reinforcement Learningcited by 20
- 2022Adversarial Cheap Talkcited by 16
- 2022Intelligent Autonomous Intersection Managementcited by 1
- 2022Security of Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 1
Built on by
Not cited yet.