# Hyperproperty-Constrained Secure Reinforcement Learning

arXiv: https://arxiv.org/abs/2508.00106 · HTML: https://mlsanity.com/paper/2508.00106/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 43 references that are in the map.
- First posted on arXiv 31 Jul 2025.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 19 of 19)

- [Omega-Regular Objectives in Model-Free Reinforcement Learning](https://mlsanity.com/paper/1810.00950/) (2018)
- [Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learning](https://mlsanity.com/paper/1909.07299/) (2019)
- [Cautious Reinforcement Learning with Logical Constraints](https://mlsanity.com/paper/2002.12156/) (2020)
- [Transfer of Temporal Logic Formulas in Reinforcement Learning](https://mlsanity.com/paper/1909.04256/) (2019)
- [Multi-Task Recommendations with Reinforcement Learning](https://mlsanity.com/paper/2302.03328/) (2023)
- [STyLuS*: A Temporal Logic Optimal Control Synthesis Algorithm for Large-Scale Multi-Robot Systems](https://mlsanity.com/paper/1809.08345/) (2018)
- [Sample Efficient Model-free Reinforcement Learning from LTL Specifications with Optimality Guarantees](https://mlsanity.com/paper/2305.01381/) (2023)
- [Reinforcement Learning with Dynamic Boltzmann Softmax Updates](https://mlsanity.com/paper/1903.05926/) (2019)
- [Policy Synthesis and Reinforcement Learning for Discounted LTL](https://mlsanity.com/paper/2305.17115/) (2023)
- [Model-Free Reinforcement Learning for Optimal Control of MarkovDecision Processes Under Signal Temporal Logic Specifications](https://mlsanity.com/paper/2109.13377/) (2021)
- [AutoHyper: Explicit-State Model Checking for HyperLTL](https://mlsanity.com/paper/2301.11229/) (2023)
- [Probabilistically Guaranteed Satisfaction of Temporal Logic Constraints During Reinforcement Learning](https://mlsanity.com/paper/2102.10063/) (2021)
- [Reinforcement Learning of Action and Query Policies with LTL Instructions under Uncertain Event Detector](https://mlsanity.com/paper/2309.02722/) (2023)
- [Time Window Temporal Logic](https://mlsanity.com/paper/1602.04294/) (2016)
- [Cybersecurity in Robotics: Challenges, Quantitative Modeling, and Practice](https://mlsanity.com/paper/2103.05789/) (2021)
- [Reinforcement Learning with Temporal Logic Constraints for Partially-Observable Markov Decision Processes](https://mlsanity.com/paper/2104.01612/) (2021)
- [STL-Based Synthesis of Feedback Controllers Using Reinforcement Learning](https://mlsanity.com/paper/2212.01022/) (2022)
- [Controller Synthesis for Hyperproperties](https://mlsanity.com/paper/2101.08880/) (2021)
- [Model Checking Time Window Temporal Logic for Hyperproperties](https://mlsanity.com/paper/2308.02554/) (2023)

## Abstract (arXiv)

Hyperproperties for Time Window Temporal Logic (HyperTWTL) is a domain-specific formal specification language known for its effectiveness in compactly representing security, opacity, and concurrency properties for robotics applications. This paper focuses on HyperTWTL-constrained secure reinforcement learning (SecRL). Although temporal logic-constrained safe reinforcement learning (SRL) is an evolving research problem with several existing literature, there is a significant research gap in exploring security-aware reinforcement learning (RL) using hyperproperties. Given the dynamics of an agent as a Markov Decision Process (MDP) and opacity/security constraints formalized as HyperTWTL, we propose an approach for learning security-aware optimal policies using dynamic Boltzmann softmax RL while satisfying the HyperTWTL constraints. The effectiveness and scalability of our proposed approach are demonstrated using a pick-up and delivery robotic mission case study. We also compare our results with two other baseline RL algorithms, showing that our proposed method outperforms them.