mlsanityData to 7 Oct 2026

Noisy Symbolic Abstractions for Deep RL: A case study with Reward Machines

arXiv 2211.10902 ↗First posted 20 Nov 2022cs.LG cs.AI cs.FLcited by 10

Not on the map. Why?

Abstract (arXiv)

Natural and formal languages provide an effective mechanism for humans to specify instructions and reward functions. We investigate how to generate policies via RL when reward functions are specified in a symbolic language captured by Reward Machines, an increasingly popular automaton-inspired structure. We are interested in the case where the mapping of environment state to a symbolic (here, Reward Machine) vocabulary -- commonly known as the labelling function -- is uncertain from the perspective of the agent. We formulate the problem of policy learning in Reward Machines with noisy symbolic abstractions as a special class of POMDP optimization problem, and investigate several methods to address the problem, building on existing and new techniques, the latter focused on predicting Reward Machine state, rather than on grounding of individual symbols. We analyze these methods and evaluate them experimentally under varying degrees of uncertainty in the correct interpretation of the symbolic vocabulary. We verify the strength of our approach and the limitation of existing methods via an empirical investigation on both illustrative, toy domains and partially observable, deep RL domains.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 34 of its 51 references that are in the map. First posted on arXiv 20 Nov 2022.

Builds on

34 of 51 references.

References · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  3. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  4. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 183
  5. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  6. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  7. 2016Reinforcement Learning With Temporal Logic Rewardscited by 114
  8. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 97
  9. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 72
  10. 2020A Composable Specification Language for Reinforcement Learning Taskscited by 68
  11. 2017Environment-Independent Task Specifications via GLTLcited by 67
  12. 2016Q-Learning for Robust Satisfaction of Signal Temporal Logic Specificationscited by 65
22 more
  1. 2021Compositional Reinforcement Learning from Logical Specificationscited by 65
  2. 2019Joint Inference of Reward Machines and Policies for Reinforcement Learningcited by 56
  3. 2018Logically-Constrained Reinforcement Learningcited by 50
  4. 2019Reinforcement Learning with Non-Markovian Rewardscited by 45
  5. 2020Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulascited by 36
  6. 2020Reward Machines for Cooperative Multi-Agent Reinforcement Learningcited by 35
  7. 2019Transfer of Temporal Logic Formulas in Reinforcement Learningcited by 31
  8. 2020Temporal-Logic-Based Reward Shaping for Continuing Reinforcement Learning Taskscited by 28
  9. 2019Modular Deep Reinforcement Learning with Temporal Logic Specificationscited by 25
  10. 2020Active Finite Reward Automaton Inference and Reinforcement Learning Using Queries and Counterexamplescited by 23
  11. 2020Reinforcement Learning Based Temporal Logic Control with Maximum Probabilistic Satisfactioncited by 22
  12. 2020Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learningcited by 21
  13. 2017A Policy Search Method For Temporal Logic Specified Reinforcement Learning Taskscited by 19
  14. 2019Induction of Subgoal Automata for Reinforcement Learningcited by 17
  15. 2021In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specificationscited by 17
  16. 2022Joint Learning of Reward Machines and Policies in Environments with Partially Known Semanticscited by 12
  17. 2022Learning to Follow Instructions in Text-Based Gamescited by 12
  18. 2019Planning With Uncertain Specifications (PUnS)cited by 11
  19. 2020Learning Non-Markovian Reward Models in MDPscited by 11
  20. 2021Lifelong Reinforcement Learning with Temporal Logic Formulas and Reward Machinescited by 5
  21. 2019Active Perception and Control from Temporal Logic Specificationscited by 4
  22. 2022LTL-Transfer: Skill Transfer for Temporal Task Specificationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 2; 2026: 3.

10 papers.

  1. 2024Reward Machines for Deep RL in Noisy and Uncertain Environmentscited by 11
  2. 2024DeepDFA: Automata Learning through Neural Probabilistic Relaxationscited by 8
  3. 2024Learning Robust Reward Machines from Noisy Labelscited by 7
  4. 2024Neural Reward Machinescited by 6
  5. 2025Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Datacited by 5
  6. 2024Learning Hidden Subgoals under Temporal Ordering Constraints in Reinforcement Learningcited by 1
  7. 2025Fully Learnable Neural Reward Machinescited by 1
  8. 2026Grounding LTL Tasks in Sub-Symbolic RL Environments for Zero-Shot Generalizationcited by 1
  9. 2026DeepDFA: Injecting Temporal Logic in Deep Learning for Sequential Subsymbolic Applicationscited by 0
  10. 2026Safe Reinforcement Learning of Autonomous Highway Driving: A Unified Framework for Safety and Efficiencycited by 0