mlsanityData to 14 Oct 2026

Neural Reward Machines

arXiv 2408.08677 ↗First posted 16 Aug 2024cs.LG cs.AIcited by 7

On the map: Deep reinforcement learning methods → Interpretable and logic-guided RL

Abstract (arXiv)

Non-markovian Reinforcement Learning (RL) tasks are very hard to solve, because agents must consider the entire history of state-action pairs to act rationally in the environment. Most works use symbolic formalisms (as Linear Temporal Logic or automata) to specify the temporally-extended task. These approaches only work in finite and discrete state environments or continuous problems for which a mapping between the raw state and a symbolic interpretation is known as a symbol grounding (SG) function. Here, we define Neural Reward Machines (NRM), an automata-based neurosymbolic framework that can be used for both reasoning and learning in non-symbolic non-markovian RL domains, which is based on the probabilistic relaxation of Moore Machines. We combine RL with semisupervised symbol grounding (SSSG) and we show that NRMs can exploit high-level symbolic knowledge in non-symbolic environments without any knowledge of the SG function, outperforming Deep RL methods which cannot incorporate prior knowledge. Moreover, we advance the research in SSSG, proposing an algorithm for analysing the groundability of temporal specifications, which is more efficient than baseline techniques of a factor 103.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods. Builds on 23 of its 36 references that are in the map. First posted on arXiv 16 Aug 2024.

Builds on

23 of 36 references.

References · 23

  1. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  3. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  4. 2019Neural Probabilistic Logic Programming in DeepProbLogcited by 394
  5. 2017A Semantic Loss Function for Deep Learning with Symbolic Knowledgecited by 270
  6. 2020Logic Tensor Networkscited by 199
  7. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 191
  8. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 74
  9. 2018Foundations for Restraining Bolts: Reinforcement Learning with LTLf/LDLf restraining specificationscited by 72
  10. 2017Environment-Independent Task Specifications via GLTLcited by 67
  11. 2019Reinforcement Learning with Non-Markovian Rewardscited by 45
  12. 2020Neural-Symbolic Integration: A Compositional Perspectivecited by 45
11 more
  1. 2021Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logiccited by 39
  2. 2020Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulascited by 36
  3. 2020Induction and Exploitation of Subgoal Automata for Reinforcement Learningcited by 26
  4. 2020Active Finite Reward Automaton Inference and Reinforcement Learning Using Queries and Counterexamplescited by 23
  5. 2020Reinforcement Learning Based Temporal Logic Control with Maximum Probabilistic Satisfactioncited by 22
  6. 2019Specification Patterns for Robotic Missionscited by 16
  7. 2022Joint Learning of Reward Machines and Policies in Environments with Partially Known Semanticscited by 12
  8. 2022Markov Abstractions for PAC Reinforcement Learning in Non-Markov Decision Processescited by 10
  9. 2022Noisy Symbolic Abstractions for Deep RL: A case study with Reward Machinescited by 10
  10. 2022Differentiable Rule Induction with Learned Relational Featurescited by 8
  11. 2021Learning Finite Linear Temporal Logic Specifications with a Specialized Neural Operatorcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025Shortcuts and Identifiability in Concept-based Models from a Neuro-Symbolic Lenscited by 13
  2. 2025Neuro-Symbolic Predictive Process Monitoringcited by 7
  3. 2025Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Datacited by 5
  4. 2026Grounding LTL Tasks in Sub-Symbolic RL Environments for Zero-Shot Generalizationcited by 1
  5. 2026DeepDFA: Injecting Temporal Logic in Deep Learning for Sequential Subsymbolic Applicationscited by 0
  6. 2026Jaxolotl: A Unified High-Performance Benchmark Suite for LTL-Based Multi-Task RLcited by 0
  7. 2026Neurosymbolic Learning of Interpretable World Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Fully Learnable Neural Reward Machines
  2. 2026Reinforcement Learning with Symbolic Reward Machines
  3. 2025Physics-Informed Reward Machines
  4. 2024Reward Machines for Deep RL in Noisy and Uncertain Environments
  5. 2025Pushdown Reward Machines for Reinforcement Learning
  6. 2025ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learning
  7. 2026Sample-Efficient Neurosymbolic Deep Reinforcement Learning
  8. 2024FORM: Learning Expressive and Transferable First-Order Logic Reward Machines