# Towards Production-Worthy Simulation for Autonomous Cyber Operations

arXiv: https://arxiv.org/abs/2508.19278 · HTML: https://mlsanity.com/paper/2508.19278/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 4 of its 16 references that are in the map.
- First posted on arXiv 23 Aug 2025.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 4 of 4)

- [Learning Cyber Defence Tactics from Scratch with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2310.05939/) (2023)
- [LLM Honeypot: Leveraging Large Language Models as Advanced Interactive Honeypot Systems](https://mlsanity.com/paper/2409.08234/) (2024)
- [Deep Reinforcement Learning for Cyber System Defense under Dynamic Adversarial Uncertainties](https://mlsanity.com/paper/2302.01595/) (2023)
- [The Path To Autonomous Cyber Defense](https://mlsanity.com/paper/2404.10788/) (2024)

## Built on by (most cited first, 2 of 2)

- [Large Language Model Integration with Reinforcement Learning to Augment Decision-Making in Autonomous Cyber Operations](https://mlsanity.com/paper/2509.05311/) (2025)
- [Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations](https://mlsanity.com/paper/2607.28826/) (2026)

## Abstract (arXiv)

Simulated environments have proven invaluable in Autonomous Cyber Operations (ACO) where Reinforcement Learning (RL) agents can be trained without the computational overhead of emulation. These environments must accurately represent cybersecurity scenarios while producing the necessary signals to support RL training. In this study, we present a framework where we first extend CybORG's Cage Challenge 2 environment by implementing three new actions: Patch, Isolate, and Unisolate, to better represent the capabilities available to human operators in real-world settings. We then propose a design for agent development where we modify the reward signals and the agent's feature space to enhance training performance. To validate these modifications, we train DQN and PPO agents in the updated environment. Our study demonstrates that CybORG can be extended with additional realistic functionality, while maintaining its ability to generate informative training signals for RL agents.