# Provable Defense against Backdoor Policies in Reinforcement Learning

arXiv: https://arxiv.org/abs/2211.10530 · HTML: https://mlsanity.com/paper/2211.10530/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 18 Nov 2022.

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks](https://mlsanity.com/paper/1805.12185/) (2018)
- [Spectral Signatures in Backdoor Attacks](https://mlsanity.com/paper/1811.00636/) (2018)
- [STRIP: A Defence Against Trojan Attacks on Deep Neural Networks](https://mlsanity.com/paper/1902.06531/) (2019)
- [Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering](https://mlsanity.com/paper/1811.03728/) (2018)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Hidden Trigger Backdoor Attacks](https://mlsanity.com/paper/1910.00033/) (2019)
- [Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks](https://mlsanity.com/paper/2101.05930/) (2021)
- [Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks](https://mlsanity.com/paper/1906.04893/) (2019)
- [Neural Trojans](https://mlsanity.com/paper/1710.00942/) (2017)
- [BACKDOORL: Backdoor Attack against Competitive Reinforcement Learning](https://mlsanity.com/paper/2105.00579/) (2021)
- [Backdoor Defense with Machine Unlearning](https://mlsanity.com/paper/2201.09538/) (2022)
- [The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Models](https://mlsanity.com/paper/2003.07233/) (2020)

## Built on by (most cited first, 18 of 18)

- [Adversarial Inception Backdoor Attacks against Reinforcement Learning](https://mlsanity.com/paper/2410.13995/) (2024)
- [Cooperative Backdoor Attack in Decentralized Reinforcement Learning with Theoretical Guarantee](https://mlsanity.com/paper/2405.15245/) (2024)
- [Fox in the Henhouse: Supply-Chain Backdoor Attacks Against Reinforcement Learning](https://mlsanity.com/paper/2505.19532/) (2025)
- [Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error](https://mlsanity.com/paper/2402.02165/) (2024)
- [Mitigating Deep Reinforcement Learning Backdoors in the Neural Activation Space](https://mlsanity.com/paper/2407.15168/) (2024)
- [Temporal Logic-Based Multi-Vehicle Backdoor Attacks against Offline RL Agents in End-to-end Autonomous Driving](https://mlsanity.com/paper/2509.16950/) (2025)
- [TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Models](https://mlsanity.com/paper/2506.12815/) (2025)
- [Belief-Enriched Pessimistic Q-Learning against Adversarial State Perturbations](https://mlsanity.com/paper/2403.04050/) (2024)
- [Backdoors in DRL: Four Environments Focusing on In-distribution Triggers](https://mlsanity.com/paper/2505.17248/) (2025)
- [Beware Untrusted Simulators -- Reward-Free Backdoor Attacks in Reinforcement Learning](https://mlsanity.com/paper/2602.05089/) (2026)
- [Reinforcement Learning-Based Approaches for Enhancing Security and Resilience in Smart Control: A Survey on Attack and Defense Methods](https://mlsanity.com/paper/2402.15617/) (2024)
- [Beyond Training-time Poisoning: Component-level and Post-training Backdoors in Deep Reinforcement Learning](https://mlsanity.com/paper/2507.04883/) (2025)
- [PolicyGuard: Towards Test-time and Step-level Adversary (Backdoor) Defense for Reinforcement Learning Agent](https://mlsanity.com/paper/2606.12896/) (2026)
- [Trading Devil RL: Backdoor attack via Stock market, Bayesian Optimization and Reinforcement Learning](https://mlsanity.com/paper/2412.17908/) (2024)
- [On the Tension Between Optimality and Adversarial Robustness in Policy Optimization](https://mlsanity.com/paper/2512.01228/) (2025)
- [BehaviorGuard: Online Backdoor Defense for Deep Reinforcement Learning](https://mlsanity.com/paper/2605.05977/) (2026)
- [Plan2Cleanse: Test-Time Backdoor Defense via Monte-Carlo Planning in Deep Reinforcement Learning](https://mlsanity.com/paper/2605.09638/) (2026)
- [ORAN-DEFEND: Subspace Detection and Sanitization of Backdoor DRL xApps in Open RAN](https://mlsanity.com/paper/2607.06647/) (2026)

## Abstract (arXiv)

We propose a provable defense mechanism against backdoor policies in reinforcement learning under subspace trigger assumption. A backdoor policy is a security threat where an adversary publishes a seemingly well-behaved policy which in fact allows hidden triggers. During deployment, the adversary can modify observed states in a particular way to trigger unexpected actions and harm the agent. We assume the agent does not have the resources to re-train a good policy. Instead, our defense mechanism sanitizes the backdoor policy by projecting observed states to a 'safe subspace', estimated from a small number of interactions with a clean (non-triggered) environment. Our sanitized policy achieves ε approximate optimality in the presence of triggers, provided the number of clean interactions is Oleft(fracD(1-γ)4 ε2right) where γ is the discounting factor and D is the dimension of state space. Empirically, we show that our sanitization defense performs well on two Atari game environments.