mlsanityData to 7 Oct 2026

Reward Shaping for Happier Autonomous Cyber Security Agents

arXiv 2310.13565 ↗First posted 20 Oct 2023cs.LG cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

As machine learning models become more capable, they have exhibited increased potential in solving complex tasks. One of the most promising directions uses deep reinforcement learning to train autonomous agents in computer network defense tasks. This work studies the impact of the reward signal that is provided to the agents when training for this task. Due to the nature of cybersecurity tasks, the reward signal is typically 1) in the form of penalties (e.g., when a compromise occurs), and 2) distributed sparsely across each defense episode. Such reward characteristics are atypical of classic reinforcement learning tasks where the agent is regularly rewarded for progress (cf. to getting occasionally penalized for failures). We investigate reward shaping techniques that could bridge this gap so as to enable agents to train more sample-efficiently and potentially converge to a better performance. We first show that deep reinforcement learning algorithms are sensitive to the magnitude of the penalties and their relative size. Then, we combine penalties with positive external rewards and study their effect compared to penalty-only training. Finally, we evaluate intrinsic curiosity as an internal positive reward mechanism and discuss why it might not be as advantageous for high-level network monitoring tasks.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 19 of its 48 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

19 of 48 references.

References · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  7. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  8. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  9. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  10. 2020The Creation and Detection of Deepfakes: A Surveycited by 246
  11. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  12. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 76
7 more
  1. 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 59
  2. 2019Autonomous Penetration Testing using Reinforcement Learningcited by 33
  3. 2021Network Environment Design for Autonomous Cyberdefensecited by 29
  4. 2021Curiosity-Driven Exploration via Latent Bayesian Surprisecited by 22
  5. 2022Beyond CAGE: Investigating Generalization of Learned Autonomous Network Defense Policiescited by 22
  6. 2023Inroads into Autonomous Network Defence using Explained Reinforcement Learningcited by 17
  7. 2023Training Automated Defense Strategies Using Graph-based Cyber Attack Simulationscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 5; 2026: 6.

19 papers.

Sort:

By area:

  1. 2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applicationscited by 34
  2. 2024Optimal Defender Strategies for CAGE-2 using Causal Modeling and Tree Searchcited by 12
  3. 2024CybORG++: An Enhanced Gym for the Development of Autonomous Cyber Agentscited by 11
  4. 2024Entity-based Reinforcement Learning for Autonomous Cyber Defencecited by 11
  5. 2024Structural Generalization in Autonomous Cyber Incident Response with Message-Passing Neural Networks and Reinforcement Learningcited by 7
  6. 2025DRMD: Deep Reinforcement Learning for Malware Detection under Concept Driftcited by 7
  7. 2024Optimizing Cyber Defense in Dynamic Active Directories through Reinforcement Learningcited by 5
  8. 2025Online Incident Response Planning under Model Misspecification through Bayesian Learning and Belief Quantizationcited by 5
  9. 2024Designing Robust Cyber-Defense Agents with Evolving Behavior Treescited by 4
  10. 2026SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecuritycited by 4
  11. 2026Beyond Rewards in Reinforcement Learning for Cyber Defencecited by 3
  12. 2026Building Better Environments for Autonomous Cyber Defencecited by 3
7 more
  1. 2025Less is more? Rewards in RL for Cyber Defencecited by 1
  2. 2025Training RL Agents for Multi-Objective Network Defense Taskscited by 1
  3. 2026STAIR: Effective Incident Response Using an End-to-End Agentic Planning Frameworkcited by 1
  4. 2024Inherently Interpretable and Uncertainty-Aware Models for Online Learning in Cyber-Security Problemscited by 0
  5. 2025Clutch Control: An Attention-based Combinatorial Bandit for Efficient Mutation in JavaScript Engine Fuzzingcited by 0
  6. 2026Learning Red Agent Policy from Observations for Neurosymbolic Autonomous Cyber Agentscited by 0
  7. 2026REPLICANT: Learning Policies for Evading and Hardening Malware Detectorscited by 0