# Plan Your Target and Learn Your Skills: Transferable State-Only Imitation Learning via Decoupled Policy Optimization

arXiv: https://arxiv.org/abs/2203.02214 · HTML: https://mlsanity.com/paper/2203.02214/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 4 Mar 2022.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 24 of 24)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving](https://mlsanity.com/paper/2010.09776/) (2020)
- [Zero-Shot Visual Imitation](https://mlsanity.com/paper/1804.08606/) (2018)
- [Recent Advances in Imitation Learning from Observation](https://mlsanity.com/paper/1905.13566/) (2019)
- [Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulation](https://mlsanity.com/paper/1703.02018/) (2017)
- [State Alignment-based Imitation Learning](https://mlsanity.com/paper/1911.10947/) (2019)
- [Off-Policy Imitation Learning from Observations](https://mlsanity.com/paper/2102.13185/) (2021)
- [Learning and Transfer of Modulated Locomotor Controllers](https://mlsanity.com/paper/1610.05182/) (2016)
- [Hierarchical Reinforcement Learning By Discovering Intrinsic Options](https://mlsanity.com/paper/2101.06521/) (2021)
- [Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic](https://mlsanity.com/paper/1901.02705/) (2019)
- [Sub-policy Adaptation for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1906.05862/) (2019)
- [Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement](https://mlsanity.com/paper/1910.04417/) (2019)
- [DAC: The Double Actor-Critic Architecture for Learning Options](https://mlsanity.com/paper/1904.12691/) (2019)
- [Energy-Based Imitation Learning](https://mlsanity.com/paper/2004.09395/) (2020)
- [Learning a Decision Module by Imitating Driver's Control Behaviors](https://mlsanity.com/paper/1912.00191/) (2019)
- [Hybrid Reinforcement Learning with Expert State Sequences](https://mlsanity.com/paper/1903.04110/) (2019)
- [Estimating Q(s,s') with Deep Deterministic Dynamics Gradients](https://mlsanity.com/paper/2002.09505/) (2020)
- [Internal Model from Observations for Reward Shaping](https://mlsanity.com/paper/1806.01267/) (2018)

## Built on by (most cited first, 12 of 12)

- [A Policy-Guided Imitation Approach for Offline Reinforcement Learning](https://mlsanity.com/paper/2210.08323/) (2022)
- [Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Function](https://mlsanity.com/paper/2302.01244/) (2023)
- [Visual Imitation Learning with Patch Rewards](https://mlsanity.com/paper/2302.00965/) (2023)
- [Diffusion Imitation from Observation](https://mlsanity.com/paper/2410.05429/) (2024)
- [Imitation Learning from Observation with Automatic Discount Scheduling](https://mlsanity.com/paper/2310.07433/) (2023)
- [TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distance](https://mlsanity.com/paper/2509.26627/) (2025)
- [Learning Rational Subgoals from Demonstrations and Instructions](https://mlsanity.com/paper/2303.05487/) (2023)
- [Controlling Large Language Model with Latent Actions](https://mlsanity.com/paper/2503.21383/) (2025)
- [NOLO: Navigate Only Look Once](https://mlsanity.com/paper/2408.01384/) (2024)
- [GAN-MPC: Training Model Predictive Controllers with Parameterized Cost Functions using Demonstrations from Non-identical Experts](https://mlsanity.com/paper/2305.19111/) (2023)
- [Planning Immediate Landmarks of Targets for Model-Free Skill Transfer across Agents](https://mlsanity.com/paper/2212.09033/) (2022)
- [Learning from Observation: A Survey of Recent Advances](https://mlsanity.com/paper/2509.19379/) (2025)

## Abstract (arXiv)

Recent progress in state-only imitation learning extends the scope of applicability of imitation learning to real-world settings by relieving the need for observing expert actions. However, existing solutions only learn to extract a state-to-action mapping policy from the data, without considering how the expert plans to the target. This hinders the ability to leverage demonstrations and limits the flexibility of the policy. In this paper, we introduce Decoupled Policy Optimization (DePO), which explicitly decouples the policy as a high-level state planner and an inverse dynamics model. With embedded decoupled policy gradient and generative adversarial training, DePO enables knowledge transfer to different action spaces or state transition dynamics, and can generalize the planner to out-of-demonstration state regions. Our in-depth experimental analysis shows the effectiveness of DePO on learning a generalized target state planner while achieving the best imitation performance. We demonstrate the appealing usage of DePO for transferring across different tasks by pre-training, and the potential for co-training agents with various skills.