# Unveiling the Black Box: A Multi-Layer Framework for Explaining Reinforcement Learning-Based Cyber Agents

arXiv: https://arxiv.org/abs/2505.11708 · HTML: https://mlsanity.com/paper/2505.11708/

## Key facts

- Cited by 4 papers in the map (data through 7 Oct 2026).
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 16 May 2025.

## Builds on (most cited first, 21 of 21)

- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [Explainable Reinforcement Learning Through a Causal Lens](https://mlsanity.com/paper/1905.10958/) (2019)
- [Non-stationary Reinforcement Learning without Prior Knowledge: An Optimal Black-box Approach](https://mlsanity.com/paper/2102.05406/) (2021)
- [How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations](https://mlsanity.com/paper/2101.08758/) (2021)
- [Generation of Policy-Level Explanations for Reinforcement Learning](https://mlsanity.com/paper/1905.12044/) (2019)
- [Developing Optimal Causal Cyber-Defence Agents via Cyber Security Simulation](https://mlsanity.com/paper/2207.12355/) (2022)
- [Inroads into Autonomous Network Defence using Explained Reinforcement Learning](https://mlsanity.com/paper/2306.09318/) (2023)
- [Learning Cyber Defence Tactics from Scratch with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2310.05939/) (2023)
- [Entity-based Reinforcement Learning for Autonomous Cyber Defence](https://mlsanity.com/paper/2410.17647/) (2024)
- [Causal Explanations for Sequential Decision-Making in Multi-Agent Systems](https://mlsanity.com/paper/2302.10809/) (2023)
- [Explaining RL Decisions with Trajectories](https://mlsanity.com/paper/2305.04073/) (2023)
- [Network Defense is Not a Game](https://mlsanity.com/paper/2104.10262/) (2021)
- [Defending Active Directory by Combining Neural Network based Dynamic Program and Evolutionary Diversity Optimisation](https://mlsanity.com/paper/2204.03397/) (2022)
- [Experiential Explanations for Reinforcement Learning](https://mlsanity.com/paper/2210.04723/) (2022)
- [Optimizing Cyber Defense in Dynamic Active Directories through Reinforcement Learning](https://mlsanity.com/paper/2406.19596/) (2024)
- [Evolving Reinforcement Learning Environment to Minimize Learner's Achievable Reward: An Application on Hardening Active Directory Systems](https://mlsanity.com/paper/2304.03998/) (2023)
- [Enhancing Network Resilience through Machine Learning-powered Graph Combinatorial Optimization: Applications in Cyber Defense and Information Diffusion](https://mlsanity.com/paper/2310.10667/) (2023)
- [CODEX: A Cluster-Based Method for Explainable Reinforcement Learning](https://mlsanity.com/paper/2312.04216/) (2023)
- [Abstracted Trajectory Visualization for Explainability in Reinforcement Learning](https://mlsanity.com/paper/2402.07928/) (2024)

## Built on by (most cited first, 4 of 4)

- [Explainable Autonomous Cyber Defense using Adversarial Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2604.04442/) (2026)
- [The Emergence of Deep Reinforcement Learning for Path Planning](https://mlsanity.com/paper/2507.15469/) (2025)
- [Task Allocation for Autonomous Machines using Computational Intelligence and Deep Reinforcement Learning](https://mlsanity.com/paper/2508.20688/) (2025)
- [Deep Reinforcement Learning for Spacecraft Attitude Control During Atmospheric Re-Entry](https://mlsanity.com/paper/2606.31291/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) agents are increasingly used to simulate sophisticated cyberattacks, but their decision-making processes remain opaque, hindering trust, debugging, and defensive preparedness. In high-stakes cybersecurity contexts, explainability is essential for understanding how adversarial strategies are formed and evolve over time. In this paper, we propose a unified, multi-layer explainability framework for RL-based attacker agents that reveals both strategic (Markov Decision Process (MDP)-level) and tactical (policy-level) reasoning. At the MDP-level, we model cyberattacks as a Partially Observable Markov Decision Process (POMDP) to expose exploration-exploitation dynamics and phase-aware behavioural shifts. At the policy-level, we analyse the temporal evolution of Q-values and use Prioritised Experience Replay (PER) to surface critical learning transitions and evolving action preferences. Evaluated across CyberBattleSim environments of increasing complexity, our framework offers interpretable insights into agent behaviour at scale. Unlike previous explainable RL methods, which are {predominantly} post-hoc, domain-specific, or limited in depth, our approach is both agent- and environment-agnostic, {supporting use cases such as red-team simulation, RL policy debugging, phase-aware threat modelling and anticipatory defence planning.} By transforming black-box learning into actionable behavioural intelligence, our framework enables both defenders and developers to better anticipate, analyse, and respond to autonomous cyber threats.