Hyperproperty-Constrained Secure Reinforcement Learning
On the map: Deep reinforcement learning methods → Safe and constrained RL
Abstract (arXiv)
Hyperproperties for Time Window Temporal Logic (HyperTWTL) is a domain-specific formal specification language known for its effectiveness in compactly representing security, opacity, and concurrency properties for robotics applications. This paper focuses on HyperTWTL-constrained secure reinforcement learning (SecRL). Although temporal logic-constrained safe reinforcement learning (SRL) is an evolving research problem with several existing literature, there is a significant research gap in exploring security-aware reinforcement learning (RL) using hyperproperties. Given the dynamics of an agent as a Markov Decision Process (MDP) and opacity/security constraints formalized as HyperTWTL, we propose an approach for learning security-aware optimal policies using dynamic Boltzmann softmax RL while satisfying the HyperTWTL constraints. The effectiveness and scalability of our proposed approach are demonstrated using a pick-up and delivery robotic mission case study. We also compare our results with two other baseline RL algorithms, showing that our proposed method outperforms them.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 43 references that are in the map. First posted on arXiv 31 Jul 2025.
Builds on
19 of 43 references.
References · 19
- 2018Omega-Regular Objectives in Model-Free Reinforcement Learningcited by 75
- 2019Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learningcited by 75
- 2020Cautious Reinforcement Learning with Logical Constraintscited by 45
- 2019Transfer of Temporal Logic Formulas in Reinforcement Learningcited by 31
- 2023Multi-Task Recommendations with Reinforcement Learningcited by 26
- 2018STyLuS*: A Temporal Logic Optimal Control Synthesis Algorithm for Large-Scale Multi-Robot Systemscited by 24
- 2023Sample Efficient Model-free Reinforcement Learning from LTL Specifications with Optimality Guaranteescited by 18
- 2019Reinforcement Learning with Dynamic Boltzmann Softmax Updatescited by 14
- 2023Policy Synthesis and Reinforcement Learning for Discounted LTLcited by 14
- 2021Model-Free Reinforcement Learning for Optimal Control of MarkovDecision Processes Under Signal Temporal Logic Specificationscited by 10
- 2023AutoHyper: Explicit-State Model Checking for HyperLTLcited by 8
- 2021Probabilistically Guaranteed Satisfaction of Temporal Logic Constraints During Reinforcement Learningcited by 6
7 more
- 2023Reinforcement Learning of Action and Query Policies with LTL Instructions under Uncertain Event Detectorcited by 5
- 2016Time Window Temporal Logiccited by 4
- 2021Cybersecurity in Robotics: Challenges, Quantitative Modeling, and Practicecited by 4
- 2021Reinforcement Learning with Temporal Logic Constraints for Partially-Observable Markov Decision Processescited by 4
- 2022STL-Based Synthesis of Feedback Controllers Using Reinforcement Learningcited by 4
- 2021Controller Synthesis for Hyperpropertiescited by 2
- 2023Model Checking Time Window Temporal Logic for Hyperpropertiescited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Shielded Reinforcement Learning Under Dynamic Temporal Logic Constraints
- 2026Integrating LTL Constraints into PPO for Safe Reinforcement Learning
- 2024Concurrent Learning of Policy and Unknown Safety Constraints in Reinforcement Learning
- 2025HypRL: Reinforcement Learning of Control Policies for Hyperproperties
- 2026PolicyGuard: Towards Test-time and Step-level Adversary (Backdoor) Defense for Reinforcement Learning Agent
- 2026Provably Safe, Yet Scalable Reinforcement Learning
- 2024Probabilistic Satisfaction of Temporal Logic Constraints in Reinforcement Learning via Adaptive Policy-Switching
- 2025Learning Probabilistic Temporal Logic Specifications for Stochastic Systems