# Policy Continuation with Hindsight Inverse Dynamics

arXiv: https://arxiv.org/abs/1910.14055 · HTML: https://mlsanity.com/paper/1910.14055/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2019.

## Built on by (most cited first, 22 of 22)

- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment](https://mlsanity.com/paper/2402.10207/) (2024)
- [Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills](https://mlsanity.com/paper/2104.07749/) (2021)
- [Rapid Exploration for Open-World Navigation with Latent Goal Models](https://mlsanity.com/paper/2104.05859/) (2021)
- [Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL](https://mlsanity.com/paper/2202.04478/) (2022)
- [Conservative Data Sharing for Multi-Task Offline Reinforcement Learning](https://mlsanity.com/paper/2109.08128/) (2021)
- [How to Leverage Unlabeled Data in Offline Reinforcement Learning](https://mlsanity.com/paper/2202.01741/) (2022)
- [Contrastive Difference Predictive Coding](https://mlsanity.com/paper/2310.20141/) (2023)
- [Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space](https://mlsanity.com/paper/2205.08129/) (2022)
- [Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of View](https://mlsanity.com/paper/2401.11237/) (2024)
- [Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Tasks](https://mlsanity.com/paper/2210.06601/) (2022)
- [Imitating Past Successes can be Very Suboptimal](https://mlsanity.com/paper/2206.03378/) (2022)
- [A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals](https://mlsanity.com/paper/2408.05804/) (2024)
- [Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond](https://mlsanity.com/paper/2310.06147/) (2023)
- [MHER: Model-based Hindsight Experience Replay](https://mlsanity.com/paper/2107.00306/) (2021)
- [Solving Minimum-Cost Reach Avoid using Reinforcement Learning](https://mlsanity.com/paper/2410.22600/) (2024)
- [Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples](https://mlsanity.com/paper/2310.07747/) (2023)
- [Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shaping](https://mlsanity.com/paper/2209.07288/) (2022)
- [Efficient Skill Discovery via Regret-Aware Optimization](https://mlsanity.com/paper/2506.21044/) (2025)
- [Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration](https://mlsanity.com/paper/2509.10656/) (2025)
- [Urban-Focused Multi-Task Offline Reinforcement Learning with Contrastive Data Sharing](https://mlsanity.com/paper/2406.14054/) (2024)
- [Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisation](https://mlsanity.com/paper/2605.13554/) (2026)

## Abstract (arXiv)

Solving goal-oriented tasks is an important but challenging problem in reinforcement learning (RL). For such tasks, the rewards are often sparse, making it difficult to learn a policy effectively. To tackle this difficulty, we propose a new approach called Policy Continuation with Hindsight Inverse Dynamics (PCHID). This approach learns from Hindsight Inverse Dynamics based on Hindsight Experience Replay, enabling the learning process in a self-imitated manner and thus can be trained with supervised learning. This work also extends it to multi-step settings with Policy Continuation. The proposed method is general, which can work in isolation or be combined with other on-policy and off-policy algorithms. On two multi-goal tasks GridWorld and FetchReach, PCHID significantly improves the sample efficiency as well as the final performance.