mlsanityData to 14 Oct 2026

A Policy-Guided Imitation Approach for Offline Reinforcement Learning

arXiv 2210.08323 ↗First posted 15 Oct 2022cs.LG cs.AIcited by 59

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Offline reinforcement learning (RL) methods can generally be categorized into two types: RL-based and Imitation-based. RL-based methods could in principle enjoy out-of-distribution generalization but suffer from erroneous off-policy evaluation. Imitation-based methods avoid off-policy evaluation but are too conservative to surpass the dataset. In this study, we propose an alternative approach, inheriting the training stability of imitation-style methods while still allowing logical out-of-distribution generalization. We decompose the conventional reward-maximizing policy in offline RL into a guide-policy and an execute-policy. During training, the guide-poicy and execute-policy are learned using only data from the dataset, in a supervised and decoupled manner. During evaluation, the guide-policy guides the execute-policy by telling where it should go so that the reward can be maximized, serving as the textitProphet. By doing so, our algorithm allows textitstate-compositionality from the dataset, rather than textitaction-compositionality conducted in prior imitation-style methods. We dumb this new approach Policy-guided Offline RL (textttPOR). textttPOR demonstrates the state-of-the-art performance on D4RL, a standard benchmark for offline RL. We also highlight the benefits of textttPOR in terms of improving with supplementary suboptimal data and easily adapting to new tasks by only changing the guide-poicy.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 56 of its 72 references that are in the map. First posted on arXiv 15 Oct 2022.

Builds on

56 of 72 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations20 Jul
  2. 2022Imitating Past Successes can be Very Suboptimal7 Jun
  3. 2022Plan Your Target and Learn Your Skills: Transferable State-Only Imitation Learning via Decoupled Policy Optimization4 Mar
  4. 2021Quantile Filtered Imitation Learning2 Dec
  5. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning27 Oct
  6. 2018Behavioral Cloning from Observation4 May

Other references · 50

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2016Layer Normalizationcited by 5,669
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  9. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  12. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
38 more
  1. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  2. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  5. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  7. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  9. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  10. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  11. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  12. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  13. 2019Learning Latent Plans from Playcited by 316
  14. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  15. 2020Critic Regularized Regressioncited by 275
  16. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  17. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 238
  18. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  19. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  20. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  21. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  22. 2021Offline RL Without Off-Policy Evaluationcited by 151
  23. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 126
  24. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  25. 2019Training Agents using Upside-Down Reinforcement Learningcited by 107
  26. 2019Reward-Conditioned Policiescited by 102
  27. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  28. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  29. 2022Learning Smooth Neural Functions via Lipschitz Regularizationcited by 86
  30. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 76
  31. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  32. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 62
  33. 2016Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Modelcited by 54
  34. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 48
  35. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  36. 2021Regularized Behavior Value Estimationcited by 32
  37. 2021Model-Based Offline Planning with Trajectory Pruningcited by 30
  38. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 23; 2025: 16; 2026: 6.

59 papers.

Sort:

By area:

  1. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  2. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  3. 2023Elastic Decision Transformercited by 67
  4. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 64
  5. 2024DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasetscited by 49
  6. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 43
  7. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  8. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 29
  9. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  10. 2025Value-guided action planning with JEPA world modelscited by 21
  11. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
  12. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gapscited by 17
38 more
  1. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
  2. 2025Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learningcited by 15
  3. 2023IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Controlcited by 14
  4. 2025Efficient Robotic Policy Learning via Latent Space Backward Planningcited by 14
  5. 2025Flow-Based Single-Step Completion for Efficient and Expressive Policy Learningcited by 14
  6. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learningcited by 10
  7. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  8. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RLcited by 10
  9. 2023PLEX: Making the Most of the Available Data for Robotic Manipulation Pretrainingcited by 9
  10. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
  11. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
  12. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  13. 2025Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperationcited by 8
  14. 2023Offline Multi-Agent Reinforcement Learning with Coupled Value Factorizationcited by 7
  15. 2024Adaptive Q-Aid for Conditional Supervised Learning in Offline Reinforcement Learningcited by 7
  16. 2025Towards Robust Zero-Shot Reinforcement Learningcited by 7
  17. 2023Offline Reinforcement Learning with Imbalanced Datasetscited by 6
  18. 2024BiRoDiff: Diffusion policies for bipedal robot locomotion on unseen terrainscited by 6
  19. 2024Predictive Coding for Decision Transformercited by 6
  20. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 6
  21. 2025Efficient Whole Slide Pathology VQA via Token Compressioncited by 6
  22. 2025Offline Goal-Conditioned Reinforcement Learning with Projective Quasimetric Planningcited by 5
  23. 2024Improving Offline Reinforcement Learning with Inaccurate Simulatorscited by 4
  24. 2024Goal-Conditioned Data Augmentation for Offline Reinforcement Learningcited by 4
  25. 2025Data Center Cooling System Optimization Using Offline Reinforcement Learningcited by 4
  26. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  27. 2025PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoningcited by 4
  28. 2024Return-Aligned Decision Transformercited by 3
  29. 2024q-exponential family for policy optimizationcited by 2
  30. 2024Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidancecited by 2
  31. 2024RGMDT: Return-Gap-Minimizing Decision Tree Extraction in Non-Euclidean Metric Spacecited by 2
  32. 2024Random Policy Enables In-Context Reinforcement Learning within Trust Horizonscited by 2
  33. 2025Offline Action-Free Learning of Ex-BMDPs by Comparing Diverse Datasetscited by 2
  34. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learningcited by 2
  35. 2026NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RLcited by 2
  36. 2024State-Constrained Offline Reinforcement Learningcited by 1
  37. 2024Strategically Conservative Q-Learningcited by 1
  38. 2025How to Provably Improve Return Conditioned Supervised Learning?cited by 1

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  2. 2024Efficient Offline Reinforcement Learning: First Imitate, then Improve
  3. 2025Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data
  4. 2021Curriculum Offline Imitation Learning
  5. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  6. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  7. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  8. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learning