# SubstratumGraphEnv: Reinforcement Learning Environment (RLE) for Modeling System Attack Paths

arXiv: https://arxiv.org/abs/2603.01340 · HTML: https://mlsanity.com/paper/2603.01340/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 12 of its 34 references that are in the map.
- First posted on arXiv 2 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 12 of 12)

- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [AndroidEnv: A Reinforcement Learning Platform for Android](https://mlsanity.com/paper/2105.13231/) (2021)
- [Network Environment Design for Autonomous Cyberdefense](https://mlsanity.com/paper/2103.07583/) (2021)
- [Graph Neural Networks based Log Anomaly Detection and Explanation](https://mlsanity.com/paper/2307.00527/) (2023)
- [Deep Reinforcement Learning for Intrusion Detection in IoT: A Survey](https://mlsanity.com/paper/2405.20038/) (2024)
- [Automating Privilege Escalation with Deep Reinforcement Learning](https://mlsanity.com/paper/2110.01362/) (2021)
- [Cloudy with a Chance of Anomalies: Dynamic Graph Neural Network for Early Detection of Cloud Services' User Anomalies](https://mlsanity.com/paper/2409.12726/) (2024)
- [Interpreting Agent Behaviors in Reinforcement-Learning-Based Cyber-Battle Simulation Platforms](https://mlsanity.com/paper/2506.08192/) (2025)
- [DRIFT: Deep Reinforcement Learning for Functional Software Testing](https://mlsanity.com/paper/2007.08220/) (2020)
- [OpenGraphGym-MG: Using Reinforcement Learning to Solve Large Graph Optimization Problems on MultiGPU Systems](https://mlsanity.com/paper/2105.08764/) (2021)
- [Reinforcement Learning for Automated Cybersecurity Penetration Testing](https://mlsanity.com/paper/2507.02969/) (2025)

## Abstract (arXiv)

Automating network security analysis, particularly the identification of potential attack paths, presents significant challenges. Due in part to the sequential, interconnected, and evolutionary nature of system events which most artificial intelligence (AI) techniques struggle to model effectively. This paper proposes a Reinforcement Learning (RL) environment generation framework that simulates the sequence of processes executed on a Windows operating system, enabling dynamic modeling of malicious processes on a system. This methodology models operating system state and transitions using a graph representation. This graph is derived from open-source System Monitor (Sysmon) logs. To address the variety in system event types, fields, and log formats, a mechanism was developed to capture and model parent-child processes from Sysmon logs. A Gymnasium environment (SubstratumGraphEnv) was constructed to establish the perceptible basis for an RL environment, and a customized PyTorch interface was also built (SubstratumBridge) to translate Gymnasium graphs into Deep Reinforcement Learning (DRL) observations and discrete actions. Graph Convolutional Networks (GCNs) concretize the graph's local and global state, which feed the distinct policy and critic heads of an Advantage Actor-Critic (A2C) model. This work's central contribution lies in the design of a novel deep graphical RL environment that automates translation of sequential user and system events, furnishing crucial context for cybersecurity analysis. This work provides a foundation for future research into shaping training parameters and advanced reward shaping, while also offering insight into which system events attributes are critical to training autonomous RL agents.