mlsanityData to 7 Oct 2026

Inverse Reinforcement Learning from a Gradient-based Learner

arXiv 2007.07812 ↗First posted 15 Jul 2020cs.LG stat.MLcited by 7

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Inverse Reinforcement Learning addresses the problem of inferring an expert's reward function from demonstrations. However, in many applications, we not only have access to the expert's near-optimal behavior, but we also observe part of her learning process. In this paper, we propose a new algorithm for this setting, in which the goal is to recover the reward function being optimized by an agent, given a sequence of policies produced during learning. Our approach is based on the assumption that the observed agent is updating her policy parameters along the gradient direction. Then we extend our method to deal with the more realistic scenario where we only have access to a dataset of learning trajectories. For both settings, we provide theoretical insights into our algorithms' performance. Finally, we evaluate the approach in a simulated GridWorld environment and on the MuJoCo environments, comparing it with the state-of-the-art baseline.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 13 of its 42 references that are in the map. First posted on arXiv 15 Jul 2020.

Builds on

13 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations12 Apr

Other references · 12

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2016OpenAI Gymcited by 1,684
  5. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  6. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  7. 2018Machine Theory of Mindcited by 222
  8. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  9. 2019Theory of Minds: Understanding Behavior in Groups Through Inverse Planningcited by 30
  10. 2019On-Policy Robot Imitation Learning from a Converging Supervisorcited by 10
  11. 2014Fast and Robust Least Squares Estimation in Corrupted Linear Modelscited by 8
  12. 2019Inverse Reinforcement Learning with Multiple Ranked Expertscited by 5

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 2; 2023: 2; 2024: 1.

7 papers.

  1. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learningcited by 16
  2. 2021Inverse Contextual Bandits: Learning How Behavior Evolves over Timecited by 12
  3. 2021Learning from an Exploring Demonstrator: Optimal Reward Estimation for Banditscited by 8
  4. 2022Modeling Human Behavior Part I -- Learning and Belief Approachescited by 1
  5. 2024Inverse Reinforcement Learning from Non-Stationary Learning Agentscited by 1
  6. 2023MERMAIDE: Learning to Align Learners using Model-Based Meta-Learningcited by 0
  7. 2023On the Effective Horizon of Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids
  2. 2021Versatile Inverse Reinforcement Learning via Cumulative Rewards
  3. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  4. 2018Teaching Inverse Reinforcement Learners via Features and Demonstrations
  5. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  6. 2022Environment Design for Inverse Reinforcement Learning
  7. 2020Inverse Reinforcement Learning via Matching of Optimality Profiles
  8. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations