mlsanityData to 7 Oct 2026

Curriculum Offline Imitation Learning

arXiv 2111.02056 ↗First posted 3 Nov 2021cs.LG cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Offline reinforcement learning (RL) tasks require the agent to learn from a pre-collected dataset with no further interactions with the environment. Despite the potential to surpass the behavioral policies, RL-based methods are generally impractical due to the training instability and bootstrapping the extrapolation errors, which always require careful hyperparameter tuning via online evaluation. In contrast, offline imitation learning (IL) has no such issues since it learns the policy directly without estimating the value function by bootstrapping. However, IL is usually limited in the capability of the behavioral policy and tends to learn a mediocre behavior from the dataset collected by the mixture of policies. In this paper, we aim to take advantage of IL but mitigate such a drawback. Observing that behavior cloning is able to imitate neighboring policies with less data, we propose textitCurriculum Offline Imitation Learning (COIL), which utilizes an experience picking strategy for imitating from adaptive neighboring policies with a higher return, and improves the current policy along curriculum stages. On continuous control benchmarks, we compare COIL against both imitation-based and RL-based methods, showing that it not only avoids just learning a mediocre behavior on mixed datasets but is also even competitive with state-of-the-art offline RL methods.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 27 references that are in the map. First posted on arXiv 3 Nov 2021.

Builds on

21 of 27 references.

Closest ancestors · 3

Same direction, newest first

  1. 2019On Value Discrepancy of Imitation Learning16 Nov
  2. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  3. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning27 Oct

Other references · 18

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  11. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  12. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
6 more
  1. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  2. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  3. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  4. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  5. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  6. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learningcited by 9

Similar, not cited

Close in wording, no citation either way

  1. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learning
  2. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  3. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  4. 2024Efficient Offline Reinforcement Learning: First Imitate, then Improve
  5. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?
  6. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  7. 2020Offline Learning from Demonstrations and Unlabeled Experience
  8. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage