mlsanityData to 7 Oct 2026

Plan Your Target and Learn Your Skills: Transferable State-Only Imitation Learning via Decoupled Policy Optimization

arXiv 2203.02214 ↗First posted 4 Mar 2022cs.LG cs.AIcited by 12

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Recent progress in state-only imitation learning extends the scope of applicability of imitation learning to real-world settings by relieving the need for observing expert actions. However, existing solutions only learn to extract a state-to-action mapping policy from the data, without considering how the expert plans to the target. This hinders the ability to leverage demonstrations and limits the flexibility of the policy. In this paper, we introduce Decoupled Policy Optimization (DePO), which explicitly decouples the policy as a high-level state planner and an inverse dynamics model. With embedded decoupled policy gradient and generative adversarial training, DePO enables knowledge transfer to different action spaces or state transition dynamics, and can generalize the planner to out-of-demonstration state regions. Our in-depth experimental analysis shows the effectiveness of DePO on learning a generalized target state planner while achieving the best imitation performance. We demonstrate the appealing usage of DePO for transferring across different tasks by pre-training, and the potential for co-training agents with various skills.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 24 of its 37 references that are in the map. First posted on arXiv 4 Mar 2022.

Builds on

24 of 37 references.

Closest ancestors · 7

Same direction, newest first

  1. 2021Off-Policy Imitation Learning from Observations25 Feb
  2. 2020Energy-Based Imitation Learning20 Apr
  3. 2019State Alignment-based Imitation Learning21 Nov
  4. 2019Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement10 Oct
  5. 2019Recent Advances in Imitation Learning from Observation30 May
  6. 2019Hybrid Reinforcement Learning with Expert State Sequences11 Mar
  7. 2018Behavioral Cloning from Observation4 May

Other references · 17

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2016The Option-Critic Architecturecited by 433
  5. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  6. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  7. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  8. 2018Zero-Shot Visual Imitationcited by 114
  9. 2017Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulationcited by 90
  10. 2016Learning and Transfer of Modulated Locomotor Controllerscited by 59
  11. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 57
  12. 2019Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Trafficcited by 50
5 more
  1. 2019Sub-policy Adaptation for Hierarchical Reinforcement Learningcited by 47
  2. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
  3. 2019Learning a Decision Module by Imitating Driver's Control Behaviorscited by 17
  4. 2020Estimating Q(s,s') with Deep Deterministic Dynamics Gradientscited by 12
  5. 2018Internal Model from Observations for Reward Shapingcited by 11

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 5; 2024: 2; 2025: 3.

12 papers.

  1. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  2. 2023Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Functioncited by 19
  3. 2023Visual Imitation Learning with Patch Rewardscited by 16
  4. 2024Diffusion Imitation from Observationcited by 12
  5. 2023Imitation Learning from Observation with Automatic Discount Schedulingcited by 10
  6. 2023Learning Rational Subgoals from Demonstrations and Instructionscited by 5
  7. 2025Controlling Large Language Model with Latent Actionscited by 5
  8. 2025TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distancecited by 5
  9. 2024NOLO: Navigate Only Look Oncecited by 3
  10. 2023GAN-MPC: Training Model Predictive Controllers with Parameterized Cost Functions using Demonstrations from Non-identical Expertscited by 1
  11. 2022Planning Immediate Landmarks of Targets for Model-Free Skill Transfer across Agentscited by 0
  12. 2025Learning from Observation: A Survey of Recent Advancescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020State-only Imitation with Transition Dynamics Mismatch
  2. 2022Imitation Learning by State-Only Distribution Matching
  3. 2018Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information
  4. 2024Efficient Imitation Learning with Conservative World Models
  5. 2022Robust Learning from Observation with Model Misspecification
  6. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitions
  7. 2022Planning for Sample Efficient Imitation Learning
  8. 2022Learning from Guided Play: Improving Exploration for Adversarial Imitation Learning with Simple Auxiliary Tasks