# Fundamental Limits of Man-in-the-Middle Attack Detection in Model-Free Reinforcement Learning

arXiv: https://arxiv.org/abs/2603.27592 · HTML: https://mlsanity.com/paper/2603.27592/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 29 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 13 of 13)

- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems](https://mlsanity.com/paper/1705.01292/) (2017)
- [Nearly Minimax Optimal Reinforcement Learning for Discounted MDPs](https://mlsanity.com/paper/2010.00587/) (2020)
- [Least-Squares Temporal Difference Learning for the Linear Quadratic Regulator](https://mlsanity.com/paper/1712.08642/) (2017)
- [Learning-based attacks in Cyber-Physical Systems: Exploration, Detection, and Control Cost trade-offs](https://mlsanity.com/paper/2011.10718/) (2020)
- [Dynamic Watermarking: Active Defense of Networked Cyber-Physical Systems](https://mlsanity.com/paper/1606.08741/) (2016)
- [Data-Injection Attacks in Stochastic Control Systems: Detectability and Performance Tradeoffs](https://mlsanity.com/paper/1704.00748/) (2017)
- [On the performance analysis of resilient networked control systems under replay attacks](https://mlsanity.com/paper/1307.2790/) (2013)
- [Detecting Integrity Attacks on Control Systems using a Moving Target Approach](https://mlsanity.com/paper/1706.08182/) (2017)
- [Tuning Windowed Chi-Squared Detectors for Sensor Attacks](https://mlsanity.com/paper/1710.02573/) (2017)
- [Statistical Watermarking for Networked Control Systems](https://mlsanity.com/paper/1709.08617/) (2017)
- [An On-line Design of Physical Watermarks](https://mlsanity.com/paper/1809.05299/) (2018)
- [Analysis of Moving Target Defense Against False Data Injection Attacks on Power Grid](https://mlsanity.com/paper/1912.11372/) (2019)

## Abstract (arXiv)

We consider the problem of learning-based man-in-the-middle (MITM) attacks in cyber-physical systems (CPS), and extend our previously proposed Bellman Deviation Detection (BDD) framework for model-free reinforcement learning (RL). We refine the standard MDP attack model by allowing the reward function to depend on both the current and subsequent states, thereby capturing reward variations induced by errors in the adversary's transition estimate. We also derive an optimal system-identification strategy for the adversary that minimizes detectable value deviations. Further, we prove that the agent's asymptotic learning time required to secure the system scales linearly with the adversary's learning time, and that this matches the optimal lower bound. Hence, the proposed detection scheme is order-optimal in detection efficiency. Finally, we extend the framework to asynchronous and intermittent attack scenarios, where reliable detection is preserved.