# Spatiotemporally Constrained Action Space Attacks on Deep Reinforcement Learning Agents

arXiv: https://arxiv.org/abs/1909.02583 · HTML: https://mlsanity.com/paper/1909.02583/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2019.

## Built on by (most cited first, 25 of 25)

- [Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learning](https://mlsanity.com/paper/2110.04471/) (2021)
- [Resilient Machine Learning for Networked Cyber Physical Systems: A Survey for Machine Learning Security to Securing Machine Learning for CPS](https://mlsanity.com/paper/2102.07244/) (2021)
- [Sparse Adversarial Attack in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2205.09362/) (2022)
- [New Challenges in Reinforcement Learning: A Survey of Security and Privacy](https://mlsanity.com/paper/2301.00188/) (2022)
- [SUB-PLAY: Adversarial Policies against Partially Observed Multi-Agent Reinforcement Learning Systems](https://mlsanity.com/paper/2402.03741/) (2024)
- [Deep-Attack over the Deep Reinforcement Learning](https://mlsanity.com/paper/2205.00807/) (2022)
- [Targeted Adversarial Attacks on Deep Reinforcement Learning Policies via Model Checking](https://mlsanity.com/paper/2212.05337/) (2022)
- [UNIDOOR: A Universal Framework for Action-Level Backdoor Attacks in Deep Reinforcement Learning](https://mlsanity.com/paper/2501.15529/) (2025)
- [Balancing detectability and performance of attacks on the control channel of Markov Decision Processes](https://mlsanity.com/paper/2109.07171/) (2021)
- [Adversarial joint attacks on legged robots](https://mlsanity.com/paper/2205.10098/) (2022)
- [Fast Certification of Vision-Language Models Using Incremental Randomized Smoothing](https://mlsanity.com/paper/2311.09024/) (2023)
- [SoK: Adversarial Machine Learning Attacks and Defences in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2301.04299/) (2023)
- [TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Models](https://mlsanity.com/paper/2506.12815/) (2025)
- [Imitating Opponent to Win: Adversarial Policy Imitation Learning in Two-player Competitive Games](https://mlsanity.com/paper/2210.16915/) (2022)
- [Security of Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2212.06123/) (2022)
- [Enhancing Security in Deep Reinforcement Learning: A Comprehensive Survey on Adversarial Attacks and Defenses](https://mlsanity.com/paper/2510.20314/) (2025)
- [Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning](https://mlsanity.com/paper/2411.13116/) (2024)
- [User-Oriented Robust Reinforcement Learning](https://mlsanity.com/paper/2202.07301/) (2022)
- [Divide and Repair: Using Options to Improve Performance of Imitation Learning Against Adversarial Demonstrations](https://mlsanity.com/paper/2306.04581/) (2023)
- [PGN: A perturbation generation network against deep reinforcement learning](https://mlsanity.com/paper/2312.12904/) (2023)
- [Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation](https://mlsanity.com/paper/2506.16753/) (2025)
- [Meta Policy Switching for Secure UAV Deconfliction in Adversarial Airspace](https://mlsanity.com/paper/2506.21127/) (2025)
- [LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchy](https://mlsanity.com/paper/2602.17312/) (2026)
- [Robust Policy Optimization via Adversarial Importance Sampling](https://mlsanity.com/paper/2609.13044/) (2026)
- [FoSeRL: Formal Sequential Robustness Certification for Reinforcement Learning Policies](https://mlsanity.com/paper/2610.04754/) (2026)

## Abstract (arXiv)

Robustness of Deep Reinforcement Learning (DRL) algorithms towards adversarial attacks in real world applications such as those deployed in cyber-physical systems (CPS) are of increasing concern. Numerous studies have investigated the mechanisms of attacks on the RL agent's state space. Nonetheless, attacks on the RL agent's action space (AS) (corresponding to actuators in engineering systems) are equally perverse; such attacks are relatively less studied in the ML literature. In this work, we first frame the problem as an optimization problem of minimizing the cumulative reward of an RL agent with decoupled constraints as the budget of attack. We propose a white-box Myopic Action Space (MAS) attack algorithm that distributes the attacks across the action space dimensions. Next, we reformulate the optimization problem above with the same objective function, but with a temporally coupled constraint on the attack budget to take into account the approximated dynamics of the agent. This leads to the white-box Look-ahead Action Space (LAS) attack algorithm that distributes the attacks across the action and temporal dimensions. Our results shows that using the same amount of resources, the LAS attack deteriorates the agent's performance significantly more than the MAS attack. This reveals the possibility that with limited resource, an adversary can utilize the agent's dynamics to malevolently craft attacks that causes the agent to fail. Additionally, we leverage these attack strategies as a possible tool to gain insights on the potential vulnerabilities of DRL agents.