mlsanityData to 7 Oct 2026

Hierarchies of Reward Machines

arXiv 2205.15752 ↗First posted 31 May 2022cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Reward machines (RMs) are a recent formalism for representing the reward function of a reinforcement learning task through a finite-state machine whose edges encode subgoals of the task using high-level events. The structure of RMs enables the decomposition of a task into simpler and independently solvable subtasks that help tackle long-horizon and/or sparse reward tasks. We propose a formalism for further abstracting the subtask structure by endowing an RM with the ability to call other RMs, thus composing a hierarchy of RMs (HRM). We exploit HRMs by treating each call to an RM as an independently solvable subtask using the options framework, and describe a curriculum-based method to learn HRMs from traces observed by the agent. Our experiments reveal that exploiting a handcrafted HRM leads to faster convergence than with a flat HRM, and that learning an HRM is feasible in cases where its equivalent flat representation is not.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 24 of its 77 references that are in the map. First posted on arXiv 31 May 2022.

Builds on

24 of 77 references.

References · 24

  1. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  2. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  3. 2016The Option-Critic Architecturecited by 433
  4. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  5. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  6. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  7. 2017Teacher-Student Curriculum Learningcited by 197
  8. 2020Prioritized Level Replaycited by 190
  9. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 183
  10. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 108
  11. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  12. 2020A Composable Specification Language for Reinforcement Learning Taskscited by 68
12 more
  1. 2019Joint Inference of Reward Machines and Policies for Reinforcement Learningcited by 56
  2. 2020A Boolean Task Algebra for Reinforcement Learningcited by 46
  3. 2019Reinforcement Learning with Non-Markovian Rewardscited by 45
  4. 2020Reward Machines for Cooperative Multi-Agent Reinforcement Learningcited by 35
  5. 2019Discovering Options for Exploration by Minimizing Cover Timecited by 30
  6. 2020Induction and Exploitation of Subgoal Automata for Reinforcement Learningcited by 26
  7. 2021The Logical Options Frameworkcited by 24
  8. 2020Active Finite Reward Automaton Inference and Reinforcement Learning Using Queries and Counterexamplescited by 23
  9. 2020Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learningcited by 21
  10. 2019Learning Compositional Neural Programs with Recursive Tree Search and Planningcited by 17
  11. 2022Joint Learning of Reward Machines and Policies in Environments with Partially Known Semanticscited by 12
  12. 2022A Hierarchical Bayesian Approach to Inverse Reinforcement Learning with Symbolic Reward Machinescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 0; 2026: 1.

4 papers.

  1. 2024Reward Machines for Deep RL in Noisy and Uncertain Environmentscited by 11
  2. 2023Learning Reward Machines in Cooperative Multi-Agent Taskscited by 2
  3. 2023Program Machine Policy: Addressing Long-Horizon Tasks by Integrating Program Synthesis and State Machinescited by 1
  4. 2026Reward Shaping and Action Masking for Compositional Tasks using Behavior Trees and LLMscited by 0