On the Effective Horizon of Inverse Reinforcement Learning
On the map: Offline RL and imitation learning from data → Inverse reinforcement learning
Abstract (arXiv)
Inverse reinforcement learning (IRL) algorithms often rely on (forward) reinforcement learning or planning, over a given time horizon, to compute an approximately optimal policy for a hypothesized reward function; they then match this policy with expert demonstrations. The time horizon plays a critical role in determining both the accuracy of reward estimates and the computational efficiency of IRL algorithms. Interestingly, an *effective time horizon* shorter than the ground-truth value often produces better results faster. This work formally analyzes this phenomenon and provides an explanation: the time horizon controls the complexity of an induced policy class and mitigates overfitting with limited data. This analysis provides a guide for the principled choice of the effective horizon for IRL. It also prompts us to re-examine the classic IRL formulation: it is more natural to learn jointly the reward and the effective horizon rather than the reward alone with a given horizon. To validate our findings, we implement a cross-validation extension and the experimental results support the theoretical analysis. The project page and code are publicly available.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 19 of its 34 references that are in the map. First posted on arXiv 13 Jul 2023.
Builds on
19 of 34 references.
Closest ancestors · 4
Same direction, newest first
- 2022Receding Horizon Inverse Reinforcement Learning9 Jun
- 2021OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching9 Sep
- 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov
- 2020Inverse Reinforcement Learning from a Gradient-based Learner15 Jul
Other references · 15
- 2016Generative Adversarial Imitation Learningcited by 1,576
- 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
- 2018An Algorithmic Perspective on Imitation Learningcited by 455
- 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
- 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
- 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
- 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
- 2019Imitation Learning as f-Divergence Minimizationcited by 121
- 2021Feedback in Imitation Learning: The Three Regimes of Covariate Shiftcited by 68
- 2020Discount Factor as a Regularizer in Reinforcement Learningcited by 44
- 2023Bridging RL Theory and Practice with the Effective Horizoncited by 31
- 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Typescited by 26
3 more
- 2022On the Role of Discount Factor in Offline Reinforcement Learningcited by 16
- 2022Spatiotemporal Costmap Inference for MPC via Deep Inverse Reinforcement Learningcited by 10
- 2021Theoretical Guarantees of Fictitious Discount Algorithms for Episodic Reinforcement Learning and Global Convergence of Policy Gradient Methodscited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
- 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
- 2024The Virtues of Pessimism in Inverse Reinforcement Learning
- 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees
- 2023Towards Theoretical Understanding of Inverse Reinforcement Learning
- 2023Inverse Reinforcement Learning without Reinforcement Learning
- 2022Active Exploration for Inverse Reinforcement Learning
- 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms