# Model Extraction Attacks Against Reinforcement Learning Based Controllers

arXiv: https://arxiv.org/abs/2304.13090 · HTML: https://mlsanity.com/paper/2304.13090/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 28 references that are in the map.
- First posted on arXiv 25 Apr 2023.

## Builds on (most cited first, 10 of 10)

- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [Stochastic Variance-Reduced Policy Gradient](https://mlsanity.com/paper/1806.05618/) (2018)
- [Random Feature-based Online Multi-kernel Learning in Environments with Unknown Dynamics](https://mlsanity.com/paper/1712.09983/) (2017)
- [Controller Synthesis with Inductive Proofs for Piecewise Linear Systems: an SMT-based Algorithm](https://mlsanity.com/paper/1509.04623/) (2015)
- [Event-Triggered State Observers for Sparse Sensor Noise/Attacks](https://mlsanity.com/paper/1309.3511/) (2013)
- [Secure State Estimation For Cyber Physical Systems Under Sensor Attacks: A Satisfiability Modulo Theory Approach](https://mlsanity.com/paper/1412.4324/) (2014)
- [PolyAR: A Highly Parallelizable Solver For Polynomial Inequality Constraints Using Convex Abstraction Refinement](https://mlsanity.com/paper/2101.04655/) (2021)
- [PolyARBerNN: A Neural Network Guided Solver and Optimizer for Bounded Polynomial Inequalities](https://mlsanity.com/paper/2204.05365/) (2022)

## Abstract (arXiv)

We introduce the problem of model-extraction attacks in cyber-physical systems in which an attacker attempts to estimate (or extract) the feedback controller of the system. Extracting (or estimating) the controller provides an unmatched edge to attackers since it allows them to predict the future control actions of the system and plan their attack accordingly. Hence, it is important to understand the ability of the attackers to perform such an attack. In this paper, we focus on the setting when a Deep Neural Network (DNN) controller is trained using Reinforcement Learning (RL) algorithms and is used to control a stochastic system. We play the role of the attacker that aims to estimate such an unknown DNN controller, and we propose a two-phase algorithm. In the first phase, also called the offline phase, the attacker uses side-channel information about the RL-reward function and the system dynamics to identify a set of candidate estimates of the unknown DNN. In the second phase, also called the online phase, the attacker observes the behavior of the unknown DNN and uses these observations to shortlist the set of final policy estimates. We provide theoretical analysis of the error between the unknown DNN and the estimated one. We also provide numerical results showing the effectiveness of the proposed algorithm.