mlsanityData to 14 Oct 2026

Reward Learning using Structural Motifs in Inverse Reinforcement Learning

arXiv 2209.13489 ↗First posted 25 Sep 2022cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

The Inverse Reinforcement Learning (textitIRL) problem has seen rapid evolution in the past few years, with important applications in domains like robotics, cognition, and health. In this work, we explore the inefficacy of current IRL methods in learning an agent's reward function from expert trajectories depicting long-horizon, complex sequential tasks. We hypothesize that imbuing IRL models with structural motifs capturing underlying tasks can enable and enhance their performance. Subsequently, we propose a novel IRL method, SMIRL, that first learns the (approximate) structure of a task as a finite-state-automaton (FSA), then uses the structural motif to solve the IRL problem. We test our model on both discrete grid world and high-dimensional continuous domain environments. We empirically show that our proposed approach successfully learns all four complex tasks, where two foundational IRL baselines fail. Our model also outperforms the baselines in sample efficiency on a simpler toy task. We further show promising test results in a modified continuous domain on tasks with compositional reward functions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 18 of its 48 references that are in the map. First posted on arXiv 25 Sep 2022.

Builds on

18 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Scalable Bayesian Inverse Reinforcement Learning12 Feb
  2. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov
  3. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress18 Jun

Other references · 15

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  4. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  5. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  6. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 191
  7. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 176
  8. 2022Adversarial Motion Priors Make Good Substitutes for Complex Reward Functionscited by 89
  9. 2016Control of Memory, Active Perception, and Action in Minecraftcited by 61
  10. 2018Optimizing Agent Behavior over Long Time Scales by Transporting Valuecited by 58
  11. 2019Joint Inference of Reward Machines and Policies for Reinforcement Learningcited by 56
  12. 2019Learning Causal State Representations of Partially Observable Environmentscited by 30
3 more
  1. 2013Learning Finite-State Controllers for Partially Observable Environmentscited by 19
  2. 2013Credal Networks under Maximum Entropycited by 13
  3. 2017Memory Augmented Control Networkscited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Active Exploration for Inverse Reinforcement Learning
  2. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations
  3. 2022Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience
  4. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  5. 2025Inversely Learning Transferable Rewards via Abstracted States
  6. 2023Towards Theoretical Understanding of Inverse Reinforcement Learning
  7. 2022Task-Guided IRL in POMDPs that Scales
  8. 2020Inverse Reinforcement Learning via Matching of Optimality Profiles