mlsanityData to 7 Oct 2026

A Hierarchical Bayesian Approach to Inverse Reinforcement Learning with Symbolic Reward Machines

arXiv 2204.09772 ↗First posted 20 Apr 2022cs.AI cs.LGcited by 9

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

A misspecified reward can degrade sample efficiency and induce undesired behaviors in reinforcement learning (RL) problems. We propose symbolic reward machines for incorporating high-level task knowledge when specifying the reward signals. Symbolic reward machines augment existing reward machine formalism by allowing transitions to carry predicates and symbolic reward outputs. This formalism lends itself well to inverse reinforcement learning, whereby the key challenge is determining appropriate assignments to the symbolic values from a few expert demonstrations. We propose a hierarchical Bayesian approach for inferring the most likely assignments such that the concretized reward machine can discriminate expert demonstrated trajectories from other trajectories with high accuracy. Experimental results show that learned reward machines can significantly improve training efficiency for complex RL tasks and generalize well across different task environment configurations.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 49 references that are in the map. First posted on arXiv 20 Apr 2022.

Builds on

26 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Active Task-Inference-Guided Deep Inverse Reinforcement Learning24 Jan

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  8. 2017Safe Reinforcement Learning via Shieldingcited by 427
  9. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  10. 2017Inverse Reward Designcited by 246
  11. 2018Programmatically Interpretable Reinforcement Learningcited by 192
  12. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 183
13 more
  1. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  2. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  3. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  4. 2016Reinforcement Learning With Temporal Logic Rewardscited by 114
  5. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  6. 2021On the Expressivity of Markov Rewardcited by 71
  7. 2021Adversarially Guided Actor-Criticcited by 47
  8. 2019An Inductive Synthesis Framework for Verifiable Reinforcement Learningcited by 39
  9. 2020Learning abstract structure for drawing by efficient motor program inductioncited by 23
  10. 2018Logically-Constrained Neural Fitted Q-Iterationcited by 17
  11. 2017Safety-Aware Apprenticeship Learningcited by 12
  12. 2021Programmatic Reward Design by Examplecited by 9
  13. 2021Program Synthesis Guided Reinforcement Learning for Partially Observed Environmentscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 3; 2025: 1; 2026: 2.

9 papers.

  1. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  2. 2024Beyond Preferences in AI Alignmentcited by 46
  3. 2024HyQE: Ranking Contexts with Hypothetical Query Embeddingscited by 6
  4. 2025Modeling Others' Minds as Codecited by 6
  5. 2022Hierarchies of Reward Machinescited by 4
  6. 2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignmentcited by 3
  7. 2026SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learningcited by 1
  8. 2023PAGAR: Taming Reward Misalignment in Inverse Reinforcement Learning-Based Imitation Learning with Protagonist Antagonist Guided Adversarial Rewardcited by 0
  9. 2026Reinforcement Learning with Symbolic Reward Machinescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Bayesian Inverse Reinforcement Learning for Non-Markovian Rewards
  2. 2020Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization
  3. 2026Nonparametric Bayesian Inverse Reinforcement Learning with Data-Parallel Gibbs Sampling
  4. 2025Learning Reward Machines from Partially Observed Policies
  5. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variables
  6. 2025Reward Compatibility: A Framework for Inverse RL
  7. 2023Distance-rank Aware Sequential Reward Learning for Inverse Reinforcement Learning with Sub-optimal Demonstrations
  8. 2018Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications