# Modeling Penetration Testing with Reinforcement Learning Using Capture-the-Flag Challenges: Trade-offs between Model-free Learning and A Priori Knowledge

arXiv: https://arxiv.org/abs/2005.12632 · HTML: https://mlsanity.com/paper/2005.12632/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2020.

## Built on by (most cited first, 3 of 3)

- [Exposing Surveillance Detection Routes via Reinforcement Learning, Attack Graphs, and Cyber Terrain](https://mlsanity.com/paper/2211.03027/) (2022)
- [Evaluation of Reinforcement Learning for Autonomous Penetration Testing using A3C, Q-learning and DQN](https://mlsanity.com/paper/2407.15656/) (2024)
- [Exploring reinforcement learning for incident response in autonomous military vehicles](https://mlsanity.com/paper/2410.21407/) (2024)

## Abstract (arXiv)

Penetration testing is a security exercise aimed at assessing the security of a system by simulating attacks against it. So far, penetration testing has been carried out mainly by trained human attackers and its success critically depended on the available expertise. Automating this practice constitutes a non-trivial problem, as the range of actions that a human expert may attempts against a system and the range of knowledge she relies on to take her decisions are hard to capture. In this paper, we focus our attention on simplified penetration testing problems expressed in the form of capture the flag hacking challenges, and we analyze how model-free reinforcement learning algorithms may help to solve them. In modeling these capture the flag competitions as reinforcement learning problems we highlight that a specific challenge that characterize penetration testing is the problem of discovering the structure of the problem at hand. We then show how this challenge may be eased by relying on different forms of prior knowledge that may be provided to the agent. In this way we demonstrate how the feasibility of tackling penetration testing using reinforcement learning may rest on a careful trade-off between model-free and model-based algorithms. By using techniques to inject a priori knowledge, we show it is possible to better direct the agent and restrict the space of its exploration problem, thus achieving solutions more efficiently.