mlsanityData to 14 Oct 2026

Expressive Temporal Specifications for Reward Monitoring

arXiv 2511.12808 ↗First posted 16 Nov 2025cs.LG cs.AI cs.LOcited by 0

On the map: Deep reinforcement learning methods → Interpretable and logic-guided RL

Abstract (arXiv)

Specifying informative and dense reward functions remains a pivotal challenge in Reinforcement Learning, as it directly affects the efficiency of agent training. In this work, we harness the expressive power of quantitative Linear Temporal Logic on finite traces ((LTLf[F])) to synthesize reward monitors that generate a dense stream of rewards for runtime-observable state trajectories. By providing nuanced feedback during training, these monitors guide agents toward optimal behaviour and help mitigate the well-known issue of sparse rewards under long-horizon decision making, which arises under the Boolean semantics dominating the current literature. Our framework is algorithm-agnostic and only relies on a state labelling function, and naturally accommodates specifying non-Markovian properties. Empirical results show that our quantitative monitors consistently subsume and, depending on the environment, outperform Boolean monitors in maximizing a quantitative measure of task completion and in reducing convergence time.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 53 references that are in the map. First posted on arXiv 16 Nov 2025.

Builds on

15 of 53 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024FORM: Learning Expressive and Transferable First-Order Logic Reward Machines31 Dec
  2. 2024Learning Robust Reward Machines from Noisy Labels27 Aug

Other references · 13

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016OpenAI Gymcited by 1,736
  3. 2017Hindsight Experience Replaycited by 1,136
  4. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  5. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 191
  6. 2016Reinforcement Learning With Temporal Logic Rewardscited by 117
  7. 2017AI Safety Gridworldscited by 110
  8. 2018Foundations for Restraining Bolts: Reinforcement Learning with LTLf/LDLf restraining specificationscited by 72
  9. 2020A Composable Specification Language for Reinforcement Learning Taskscited by 69
  10. 2014A Learning Based Approach to Control Synthesis of Markov Decision Processes for Linear Temporal Logic Specificationscited by 49
  11. 2023Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approachcited by 19
  12. 2023Counting Reward Automata: Sample Efficient Reinforcement Learning Through the Exploitation of Reward Function Structurecited by 6
1 more
  1. 2021Being correct is not enough: efficient verification using robust linear temporal logiccited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Expressive Reward Synthesis with the Runtime Monitoring Language
  2. 2026Do It for HER: First-Order Temporal Logic Reward Specification in Reinforcement Learning (Extended Version)
  3. 2024Adaptive Reward Design for Reinforcement Learning
  4. 2024Monitored Markov Decision Processes
  5. 2024Logical Specifications-guided Dynamic Task Sampling for Reinforcement Learning Agents
  6. 2024Directed Exploration in Reinforcement Learning from Linear Temporal Logic
  7. 2025About Time: Model-free Reinforcement Learning with Timed Reward Machines
  8. 2026Live LTL Progress Tracking: Towards Task-Based Exploration