Offline Imitation Learning by Controlling the Effective Planning Horizon
On the map: Offline RL and imitation learning from data → Imitation learning methods
Abstract (arXiv)
In offline imitation learning (IL), we generally assume only a handful of expert trajectories and a supplementary offline dataset from suboptimal behaviors to learn the expert policy. While it is now common to minimize the divergence between state-action visitation distributions so that the agent also considers the future consequences of an action, a sampling error in an offline dataset may lead to erroneous estimates of state-action visitations in the offline case. In this paper, we investigate the effect of controlling the effective planning horizon (i.e., reducing the discount factor) as opposed to imposing an explicit regularizer, as previously studied. Unfortunately, it turns out that the existing algorithms suffer from magnified approximation errors when the effective planning horizon is shortened, which results in a significant degradation in performance. We analyze the main cause of the problem and provide the right remedies to correct the algorithm. We show that the corrected algorithm improves on popular imitation learning benchmarks by controlling the effective planning horizon rather than an explicit regularization.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 15 of its 23 references that are in the map. First posted on arXiv 18 Jan 2024.
Builds on
15 of 23 references.
Closest ancestors · 7
Same direction, newest first
- 2022Rethinking ValueDice: Does It Really Improve Performance?5 Feb
- 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching4 Feb
- 2020Error Bounds of Imitating Policies and Environments22 Oct
- 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
- 2019Imitation Learning as f-Divergence Minimization30 May
- 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep
- 2018Generative Adversarial Imitation from Observation17 Jul
Other references · 8
- 2017Improved Training of Wasserstein GANscited by 2,223
- 2016Generative Adversarial Imitation Learningcited by 1,576
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
- 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
- 2019Behavior Regularized Offline Reinforcement Learningcited by 654
- 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
- 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 22
- 2022On the Role of Discount Factor in Offline Reinforcement Learningcited by 16
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
- 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
- 2023A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories
- 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations
- 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
- 2024Offline Imitation Learning with Model-based Reverse Augmentation
- 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning
- 2024How to Leverage Diverse Demonstrations in Offline Imitation Learning