# Beyond Rewards in Reinforcement Learning for Cyber Defence

arXiv: https://arxiv.org/abs/2602.04809 · HTML: https://mlsanity.com/paper/2602.04809/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 4 Feb 2026.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 23 of 23)

- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Learning by Playing - Solving Sparse Reward Tasks from Scratch](https://mlsanity.com/paper/1802.10567/) (2018)
- [Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping](https://mlsanity.com/paper/2011.02669/) (2020)
- [CybORG: A Gym for the Development of Autonomous Cyber Agents](https://mlsanity.com/paper/2108.09118/) (2021)
- [Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)](https://mlsanity.com/paper/2112.15422/) (2021)
- [Measuring the Reliability of Reinforcement Learning Algorithms](https://mlsanity.com/paper/1912.05663/) (2019)
- [When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Search](https://mlsanity.com/paper/2406.08705/) (2024)
- [Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Risk](https://mlsanity.com/paper/2206.04436/) (2022)
- [Developing Optimal Causal Cyber-Defence Agents via Cyber Security Simulation](https://mlsanity.com/paper/2207.12355/) (2022)
- [On Autonomous Agents in a Cyber Defence Environment](https://mlsanity.com/paper/2309.07388/) (2023)
- [Automated Cyber Defence: A Review](https://mlsanity.com/paper/2303.04926/) (2023)
- [Reward Shaping for Happier Autonomous Cyber Security Agents](https://mlsanity.com/paper/2310.13565/) (2023)
- [Inroads into Autonomous Network Defence using Explained Reinforcement Learning](https://mlsanity.com/paper/2306.09318/) (2023)
- [Hierarchical Multi-agent Reinforcement Learning for Cyber Network Defense](https://mlsanity.com/paper/2410.17351/) (2024)
- [Finding Needles in a Moving Haystack: Prioritizing Alerts with Adversarial Reinforcement Learning](https://mlsanity.com/paper/1906.08805/) (2019)
- [Efficient Off-Policy Safe Reinforcement Learning Using Trust Region Conditional Value at Risk](https://mlsanity.com/paper/2312.00342/) (2023)
- [CybORG++: An Enhanced Gym for the Development of Autonomous Cyber Agents](https://mlsanity.com/paper/2410.16324/) (2024)
- [Entity-based Reinforcement Learning for Autonomous Cyber Defence](https://mlsanity.com/paper/2410.17647/) (2024)
- [AutoCAT: Reinforcement Learning for Automated Exploration of Cache-Timing Attacks](https://mlsanity.com/paper/2208.08025/) (2022)
- [Canaries and Whistles: Resilient Drone Communication Networks with (or without) Deep Reinforcement Learning](https://mlsanity.com/paper/2312.04940/) (2023)

## Built on by (most cited first, 3 of 3)

- [SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecurity](https://mlsanity.com/paper/2602.08690/) (2026)
- [Building Better Environments for Autonomous Cyber Defence](https://mlsanity.com/paper/2604.08805/) (2026)
- [Measuring Security Without Fooling Ourselves: Why Benchmarking Agents Is Hard](https://mlsanity.com/paper/2605.22568/) (2026)

## Abstract (arXiv)

Recent years have seen an explosion of interest in autonomous cyber defence agents trained to defend computer networks using deep reinforcement learning. These agents are typically trained in cyber gym environments using dense, highly engineered reward functions which combine many penalties and incentives for a range of (un)desirable states and costly actions. Dense rewards help alleviate the challenge of exploring complex environments but risk biasing agents towards suboptimal and potentially riskier solutions, a critical issue in complex cyber environments. We thoroughly evaluate the impact of reward function structure on learning and policy behavioural characteristics using a variety of sparse and dense reward functions, two well-established cyber gyms, a range of network sizes, and both policy gradient and value-based RL algorithms. Our evaluation is enabled by a novel ground truth evaluation approach which allows directly comparing between different reward functions, illuminating the nuanced inter-relationships between rewards, action space and the risks of suboptimal policies in cyber environments. Our results show that sparse rewards, provided they are goal aligned and can be encountered frequently, uniquely offer both enhanced training reliability and more effective cyber defence agents with lower-risk policies. Surprisingly, sparse rewards can also yield policies that are better aligned with cyber defender goals and make sparing use of costly defensive actions without explicit reward-based numerical penalties.