mlsanityData to 14 Oct 2026

HypRL: Reinforcement Learning of Control Policies for Hyperproperties

arXiv 2504.04675 ↗First posted 7 Apr 2025cs.AI cs.LOcited by 2

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reward shaping in multi-agent reinforcement learning (MARL) for complex tasks remains a significant challenge. Existing approaches often fail to find optimal solutions or cannot efficiently handle such tasks. We propose HYPRL, a specification-guided reinforcement learning framework that learns control policies w.r.t. hyperproperties expressed in HyperLTL. Hyperproperties constitute a powerful formalism for specifying objectives and constraints over sets of execution traces across agents. To learn policies that maximize the satisfaction of a HyperLTL formula φ, we apply Skolemization to manage quantifier alternations and define quantitative robustness functions to shape rewards over execution traces of a Markov decision process with unknown transitions. A suitable RL algorithm is then used to learn policies that collectively maximize the expected reward and, consequently, increase the probability of satisfying φ. We evaluate HYPRL on a diverse set of benchmarks, including safety-aware planning, Deep Sea Treasure, and the Post Correspondence Problem. We also compare with specification-driven baselines to demonstrate the effectiveness and efficiency of HYPRL.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 38 of its 66 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

38 of 66 references.

References · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Safe Reinforcement Learning via Shieldingcited by 449
  3. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 203
  4. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 191
  5. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 177
  6. 2016Reinforcement Learning With Temporal Logic Rewardscited by 117
  7. 2019Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guaranteescited by 77
  8. 2018Foundations for Restraining Bolts: Reinforcement Learning with LTLf/LDLf restraining specificationscited by 72
  9. 2020A Composable Specification Language for Reinforcement Learning Taskscited by 69
  10. 2016Q-Learning for Robust Satisfaction of Signal Temporal Logic Specificationscited by 67
  11. 2021Compositional Reinforcement Learning from Logical Specificationscited by 67
  12. 2021Safe Multi-Agent Reinforcement Learning via Shieldingcited by 61
26 more
  1. 2018Multi-Agent Reinforcement Learning Based Resource Allocation for UAV Networkscited by 56
  2. 2018Logically-Constrained Reinforcement Learningcited by 51
  3. 2019Learning Fairness in Multi-Agent Systemscited by 44
  4. 2020Learning Fair Policies in Decentralized Cooperative Multi-Agent Reinforcement Learningcited by 40
  5. 2020Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulascited by 36
  6. 2019Transfer of Temporal Logic Formulas in Reinforcement Learningcited by 31
  7. 2021Multi-Agent Reinforcement Learning with Temporal Logic Specificationscited by 27
  8. 2019Modular Deep Reinforcement Learning with Temporal Logic Specificationscited by 25
  9. 2022Online Shielding for Reinforcement Learningcited by 22
  10. 2014Temporal Logics for Hyperpropertiescited by 19
  11. 2017Programmable Agentscited by 14
  12. 2024Reinforcement Learning with LTL and ω-Regular Objectives via Optimality-Preserving Translation to Average Rewardscited by 14
  13. 2020Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learningcited by 9
  14. 2022Specification-Guided Learning of Nash Equilibria with High Social Welfarecited by 8
  15. 2019Fairness in Multi-agent Reinforcement Learning for Stock Tradingcited by 5
  16. 2021A Temporal Logic for Strategic Hyperpropertiescited by 5
  17. 2024Cooperation and Fairness in Multi-Agent Reinforcement Learningcited by 5
  18. 2024Non-Deterministic Planning for Hyperproperty Verificationcited by 4
  19. 2024Guiding Multi-agent Multi-task Reinforcement Learning by a Hierarchical Framework with Logical Reward Shapingcited by 4
  20. 2016Deciding Hyperpropertiescited by 3
  21. 2021The Complexity of Monitoring Hyperpropertiescited by 3
  22. 2022Logic-based Reward Shaping for Multi-Agent Reinforcement Learningcited by 3
  23. 2023On Alternating-Time Temporal Logic, Hyperproperties, and Strategy Sharingcited by 2
  24. 2022HyperATL*: A Logic for Hyperproperties in Multi-Agent Systemscited by 1
  25. 2024Hyper Strategy Logiccited by 1
  26. 2024Tracy, Traces, and Transducers: Computable Counterexamples and Explanations for HyperLTL Model-Checkingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026HyPOLE: Hyperproperty-Guided Multi-Agent Reinforcement Learning under Partial Observationcited by 0
  2. 2026Multi-Agent Planning with Spatio-Temporal and Topological Constraints using STL-GOcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Logical Specifications-guided Dynamic Task Sampling for Reinforcement Learning Agents
  2. 2025Automating the Refinement of Reinforcement Learning Specifications
  3. 2025Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learning
  4. 2026SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learning
  5. 2024Effective Reward Specification in Deep Reinforcement Learning
  6. 2024Combining Automated Optimisation of Hyperparameters and Reward Shape
  7. 2025Provably Optimal Reinforcement Learning under Safety Filtering
  8. 2026SEARL: Joint Optimization of Policy and Tool Graph Memory for Self-Evolving Agents