mlsanityData to 14 Oct 2026

Fully Learnable Neural Reward Machines

arXiv 2509.19017 ↗First posted 23 Sep 2025cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → Interpretable and logic-guided RL

Abstract (arXiv)

Non-Markovian Reinforcement Learning (RL) tasks present significant challenges, as agents must reason over entire trajectories of state-action pairs to make optimal decisions. A common strategy to address this is through symbolic formalisms, such as Linear Temporal Logic (LTL) or automata, which provide a structured way to express temporally extended objectives. However, these approaches often rely on restrictive assumptions -- such as the availability of a predefined Symbol Grounding (SG) function mapping raw observations to high-level symbolic representations, or prior knowledge of the temporal task. In this work, we propose a fully learnable version of Neural Reward Machines (NRM), which can learn both the SG function and the automaton end-to-end, removing any reliance on prior knowledge. Our approach is therefore as easily applicable as classic deep RL (DRL) approaches, while being far more explainable, because of the finite and compact nature of automata. Furthermore, we show that by integrating Fully Learnable Reward Machines (FLNRM) with DRL, our method outperforms previous approaches based on Recurrent Neural Networks (RNNs).

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 20 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

15 of 20 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Detecting Hidden Triggers: Mapping Non-Markov Reward Functions to Markov20 Jan

Other references · 14

  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  2. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  3. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 191
  4. 2018Foundations for Restraining Bolts: Reinforcement Learning with LTLf/LDLf restraining specificationscited by 72
  5. 2017Environment-Independent Task Specifications via GLTLcited by 67
  6. 2019Reinforcement Learning with Non-Markovian Rewardscited by 45
  7. 2020Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulascited by 36
  8. 2020Induction and Exploitation of Subgoal Automata for Reinforcement Learningcited by 26
  9. 2020Active Finite Reward Automaton Inference and Reinforcement Learning Using Queries and Counterexamplescited by 23
  10. 2020Reinforcement Learning Based Temporal Logic Control with Maximum Probabilistic Satisfactioncited by 22
  11. 2019Specification Patterns for Robotic Missionscited by 16
  12. 2022Joint Learning of Reward Machines and Policies in Environments with Partially Known Semanticscited by 12
2 more
  1. 2022Markov Abstractions for PAC Reinforcement Learning in Non-Markov Decision Processescited by 10
  2. 2022Noisy Symbolic Abstractions for Deep RL: A case study with Reward Machinescited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Grounding LTL Tasks in Sub-Symbolic RL Environments for Zero-Shot Generalizationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Neural Reward Machines
  2. 2026Reinforcement Learning with Symbolic Reward Machines
  3. 2025About Time: Model-free Reinforcement Learning with Timed Reward Machines
  4. 2024FORM: Learning Expressive and Transferable First-Order Logic Reward Machines
  5. 2024Reward Machines for Deep RL in Noisy and Uncertain Environments
  6. 2025Pushdown Reward Machines for Reinforcement Learning
  7. 2025Physics-Informed Reward Machines
  8. 2025ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learning