mlsanityData to 14 Oct 2026

OLLIE: Imitation Learning from Offline Pretraining to Online Finetuning

arXiv 2405.17477 ↗First posted 24 May 2024cs.LG cs.AIcited by 6

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

In this paper, we study offline-to-online Imitation Learning (IL) that pretrains an imitation policy from static demonstration data, followed by fast finetuning with minimal environmental interaction. We find the na"ive combination of existing offline IL and online IL methods tends to behave poorly in this context, because the initial discriminator (often used in online IL) operates randomly and discordantly against the policy initialization, leading to misguided policy optimization and textitunlearning of pretraining knowledge. To overcome this challenge, we propose a principled offline-to-online IL method, named textttOLLIE, that simultaneously learns a near-expert policy initialization along with an textitaligned discriminator initialization, which can be seamlessly integrated into online IL, achieving smooth and fast finetuning. Empirically, textttOLLIE consistently and significantly outperforms the baseline methods in 20 challenging tasks, from continuous control to vision-based domains, in terms of performance, demonstration efficiency, and convergence speed. This work may serve as a foundation for further exploration of pretraining and finetuning in the context of IL.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 53 of its 101 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

53 of 101 references.

Closest ancestors · 23

Same direction, newest first

  1. 2024A Survey of Imitation Learning Methods, Environments and Metrics30 Apr
  2. 2023Coherent Soft Imitation Learning25 May
  3. 2023Get Back Here: Robust Imitation by Return-to-Distribution Planning2 May
  4. 2022Proximal Point Imitation Learning22 Sep
  5. 2022Understanding Adversarial Imitation Learning in Small Sample Regime: A Stage-coupled Analysis3 Aug
  6. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations20 Jul
  7. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching4 Feb
  8. 2021A Critique of Strictly Batch Imitation Learning5 Oct
  9. 2021Implicit Behavioral Cloning1 Sep
  10. 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
  11. 2021SoftDICE for Imitation Learning: Rethinking Off-policy Distribution Matching6 Jun
  12. 2021What Matters for Adversarial Imitation Learning?1 Jun
11 more
  1. 2020Offline Learning from Demonstrations and Unlabeled Experience27 Nov
  2. 2020Toward the Fundamental Limits of Imitation Learning13 Sep
  3. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matching25 Jun
  4. 2020When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence24 Jun
  5. 2020Primal Wasserstein Imitation Learning8 Jun
  6. 2020Augmenting GAIL with BC for sample efficient imitation learning21 Jan
  7. 2020On Computation and Generalization of Generative Adversarial Imitation Learning9 Jan
  8. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  9. 2019Goal-conditioned Imitation Learning13 Jun
  10. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards27 May
  11. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2016Generative Adversarial Imitation Learningcited by 1,597
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  8. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  9. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  10. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  11. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  12. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
18 more
  1. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  2. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  3. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  4. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  5. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
  6. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 62
  7. 2021Scalable Bayesian Inverse Reinforcement Learningcited by 56
  8. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guaranteescited by 42
  9. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  10. 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learningcited by 29
  11. 2020Generalization Bounds for Stochastic Saddle Point Problemscited by 28
  12. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  13. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 23
  14. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  15. 2023CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learningcited by 20
  16. 2017InfoGAIL: Interpretable Imitation Learning from Visual Demonstrationscited by 11
  17. 2023Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Datacited by 11
  18. 2023Warm-Start Actor-Critic: From Approximation Error to Sub-optimality Gapcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 2.

6 papers.

  1. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  2. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuningcited by 3
  3. 2025Learning When to Quit in Sales Conversationscited by 2
  4. 2024SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidancecited by 0
  5. 2026Latent Wasserstein Adversarial Imitation Learningcited by 0
  6. 2026Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
  2. 2025Offline Imitation Learning upon Arbitrary Demonstrations by Pre-Training Dynamics Representations
  3. 2025A Recipe for Efficient Sim-to-Real Transfer in Manipulation with Online Imitation-Pretrained World Models
  4. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
  5. 2025Robust Offline Imitation Learning Through State-level Trajectory Stitching
  6. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations
  7. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  8. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learning