mlsanityData to 7 Oct 2026

Automating the Refinement of Reinforcement Learning Specifications

arXiv 2512.01047 ↗First posted 30 Nov 2025cs.AI cs.LG cs.ROcited by 0

Not on the map. Why?

Abstract (arXiv)

Logical specifications have been shown to help reinforcement learning algorithms in achieving complex tasks. However, when a task is under-specified, agents might fail to learn useful policies. In this work, we explore the possibility of improving coarse-grained logical specifications via an exploration-guided strategy. We propose AutoSpec, a framework that searches for a logical specification refinement whose satisfaction implies satisfaction of the original specification, but which provides additional guidance therefore making it easier for reinforcement learning algorithms to learn useful policies. AutoSpec is applicable to reinforcement learning tasks specified via the SpectRL specification logic. We exploit the compositional nature of specifications written in SpectRL, and design four refinement procedures that modify the abstract graph of the specification by either refining its existing edge specifications or by introducing new edge specifications. We prove that all four procedures maintain specification soundness, i.e. any trajectory satisfying the refined specification also satisfies the original. We then show how AutoSpec can be integrated with existing reinforcement learning algorithms for learning policies from logical specifications. Our experiments demonstrate that AutoSpec yields promising improvements in terms of the complexity of control tasks that can be solved, when refined logical specifications produced by AutoSpec are utilized.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 30 Nov 2025.

Builds on

23 of 37 references.

References · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  3. 2016Reinforcement Learning With Temporal Logic Rewardscited by 114
  4. 2019Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guaranteescited by 77
  5. 2018Omega-Regular Objectives in Model-Free Reinforcement Learningcited by 74
  6. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 72
  7. 2021On the Expressivity of Markov Rewardcited by 71
  8. 2018Foundations for Restraining Bolts: Reinforcement Learning with LTLf/LDLf restraining specificationscited by 69
  9. 2020A Composable Specification Language for Reinforcement Learning Taskscited by 68
  10. 2016Q-Learning for Robust Satisfaction of Signal Temporal Logic Specificationscited by 65
  11. 2021Compositional Reinforcement Learning from Logical Specificationscited by 65
  12. 2021panda-gym: Open-source goal-conditioned environments for robotic learningcited by 56
11 more
  1. 2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applicationscited by 34
  2. 2019Transfer of Temporal Logic Formulas in Reinforcement Learningcited by 31
  3. 2021Stability Verification in Stochastic Control Systems via Neural Network Supermartingalescited by 24
  4. 2022Learning Control Policies for Stochastic Systems with Reach-avoid Guaranteescited by 21
  5. 2024DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RLcited by 20
  6. 2021Verifiable and Compositional Reinforcement Learning Systemscited by 12
  7. 2023Compositional Policy Learning in Stochastic Control Systems with Formal Guaranteescited by 12
  8. 2022LCRL: Certified Policy Synthesis via Logically-Constrained Reinforcement Learningcited by 11
  9. 2024Logical Specifications-guided Dynamic Task Sampling for Reinforcement Learning Agentscited by 3
  10. 2015On the Hardness of Almost-Sure Terminationcited by 2
  11. 2023Verifiable Reinforcement Learning Systems via Compositionalitycited by 2

Built on by

Not cited yet.