On the Perturbed States for Transformed Input-robust Reinforcement Learning

arXiv 2408.00023 ↗First posted 31 Jul 2024cs.LGcited by 0

On the map: Deep reinforcement learning methods → Adversarial robustness and RL in finance

Abstract (arXiv)

Reinforcement Learning (RL) agents demonstrating proficiency in a training environment exhibit vulnerability to adversarial perturbations in input observations during deployment. This underscores the importance of building a robust agent before its real-world deployment. To alleviate the challenging point, prior works focus on developing robust training-based procedures, encompassing efforts to fortify the deep neural network component's robustness or subject the agent to adversarial training against potent attacks. In this work, we propose a novel method referred to as Transformed Input-robust RL (TIRL), which explores another avenue to mitigate the impact of adversaries by employing input transformation-based defenses. Specifically, we introduce two principles for applying transformation-based defenses in learning robust RL agents: (1) autoencoder-styled denoising to reconstruct the original state and (2) bounded transformations (bit-depth reduction and vector quantization (VQ)) to achieve close transformed inputs. The transformations are applied to the state before feeding it into the policy network. Extensive experiments on multiple MuJoCo environments demonstrate that input transformation-based defenses, i.e., VQ, defend against several adversaries in the state observations. The official code is available at https://github.com/tunglm2203/tirl

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods. Builds on 46 of its 62 references that are in the map. First posted on arXiv 31 Jul 2024.

Builds on

46 of 62 references.

References · 46

2014Adam: A Method for Stochastic Optimizationcited by 37,240
2020Denoising Diffusion Probabilistic Modelscited by 21,076
2017Proximal Policy Optimization Algorithmscited by 15,215
2014Explaining and Harnessing Adversarial Examplescited by 7,364
2022Generative Adversarial Networkscited by 5,627
2013Intriguing properties of neural networkscited by 5,155
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
2015Continuous control with deep reinforcement learningcited by 3,407
2015Trust Region Policy Optimizationcited by 2,580
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
2016OpenAI Gymcited by 1,736

34 more

2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
2017Constrained Policy Optimizationcited by 983
2016Adversarial Machine Learning at Scalecited by 936
2017Countering Adversarial Images using Input Transformationscited by 520
2017Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networkscited by 486
2022Diffusion Models for Adversarial Purificationcited by 476
2018Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Modelscited by 362
2017Defense against Adversarial Attacks Using High-Level Representation Guided Denoisercited by 329
2017Adversarial Attacks on Neural Network Policiescited by 320
2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 225
2016A study of the effect of JPG compression on adversarial imagescited by 197
2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 166
2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 154
2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 135
2017Delving into adversarial attacks on deep policiescited by 104
2017Vulnerability of Deep Reinforcement Learning to Policy Induction Attackscited by 91
2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 84
2017Virtual to Real Reinforcement Learning for Autonomous Drivingcited by 80
2020Robust Deep Reinforcement Learning through Adversarial Losscited by 75
2018Deflecting Adversarial Attacks with Pixel Deflectioncited by 71
2020Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learningcited by 63
2017APE-GAN: Adversarial Perturbation Elimination with GANcited by 57
2021Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RLcited by 55
2017NO Need to Worry about Adversarial Examples in Object Detection in Autonomous Vehiclescited by 48
2022Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learningcited by 43
2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
2017Whatever Does Not Kill Deep Reinforcement Learning, Makes It Strongercited by 35
2019Online Robustness Training for Deep Reinforcement Learningcited by 26
2019Optimal Attacks on Reinforcement Learning Policiescited by 21
2022Robust Deep Reinforcement Learning through Bootstrapped Opportunistic Curriculumcited by 15
2021Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environmentcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way