mlsanityData to 7 Oct 2026

Accelerating Inverse Reinforcement Learning with Expert Bootstrapping

arXiv 2402.02608 ↗First posted 4 Feb 2024cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Existing inverse reinforcement learning methods (e.g. MaxEntIRL, f-IRL) search over candidate reward functions and solve a reinforcement learning problem in the inner loop. This creates a rather strange inversion where a harder problem, reinforcement learning, is in the inner loop of a presumably easier problem, imitation learning. In this work, we show that better utilization of expert demonstrations can reduce the need for hard exploration in the inner RL loop, hence accelerating learning. Specifically, we propose two simple recipes: (1) placing expert transitions into the replay buffer of the inner RL algorithm (e.g. Soft-Actor Critic) which directly informs the learner about high reward states instead of forcing the learner to discover them through extensive exploration, and (2) using expert actions in Q value bootstrapping in order to improve the target Q value estimates and more accurately describe high value expert states. Our methods show significant gains over a MaxEntIRL baseline on the benchmark MuJoCo suite of tasks, speeding up recovery to 70% of deterministic expert performance by 2.13x on HalfCheetah-v2, 2.6x on Ant-v2, 18x on Hopper-v2, and 3.36x on Walker2d-v2.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 13 of its 22 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

13 of 22 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Scalable Bayesian Inverse Reinforcement Learning12 Feb
  2. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov

Other references · 11

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  6. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  7. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  8. 2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learningcited by 39
  9. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewardscited by 28
  10. 2018Fast Policy Learning through Imitation and Reinforcementcited by 26
  11. 2016Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replaycited by 22

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Hybrid Inverse Reinforcement Learning
  2. 2023Inverse Reinforcement Learning without Reinforcement Learning
  3. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  4. 2022Active Exploration for Inverse Reinforcement Learning
  5. 2024The Virtues of Pessimism in Inverse Reinforcement Learning
  6. 2026Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching
  7. 2020Inverse Reinforcement Learning from a Gradient-based Learner
  8. 2023Imitation Bootstrapped Reinforcement Learning