# Backward Learning for Goal-Conditioned Policies

arXiv: https://arxiv.org/abs/2312.05044 · HTML: https://mlsanity.com/paper/2312.05044/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 8 Dec 2023.

## Builds on (most cited first, 25 of 25)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning](https://mlsanity.com/paper/1910.11956/) (2019)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Transformer-based World Models Are Happy With 100k Interactions](https://mlsanity.com/paper/2303.07109/) (2023)
- [Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills](https://mlsanity.com/paper/2104.07749/) (2021)
- [Imitation Learning from Imperfect Demonstration](https://mlsanity.com/paper/1901.09387/) (2019)
- [Deep Hierarchical Planning from Pixels](https://mlsanity.com/paper/2206.04114/) (2022)
- [Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning](https://mlsanity.com/paper/2205.11790/) (2022)
- [Offline Reinforcement Learning with Reverse Model-based Imagination](https://mlsanity.com/paper/2110.00188/) (2021)
- [Bidirectional Model-based Policy Optimization](https://mlsanity.com/paper/2007.01995/) (2020)
- [Recall Traces: Backtracking Models for Efficient Reinforcement Learning](https://mlsanity.com/paper/1804.00379/) (2018)
- [Forward-Backward Reinforcement Learning](https://mlsanity.com/paper/1803.10227/) (2018)
- [Forethought and Hindsight in Credit Assignment](https://mlsanity.com/paper/2010.13685/) (2020)
- [Backward Imitation and Forward Reinforcement Learning via Bi-directional Model Rollouts](https://mlsanity.com/paper/2208.02434/) (2022)
- [Time-Myopic Go-Explore: Learning A State Representation for the Go-Explore Paradigm](https://mlsanity.com/paper/2301.05635/) (2023)

## Built on by (most cited first, 2 of 2)

- [Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets](https://mlsanity.com/paper/2406.01150/) (2024)
- [World-Model Policy Arbiter for Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2610.10932/) (2026)

## Abstract (arXiv)

Can we learn policies in reinforcement learning without rewards? Can we learn a policy just by trying to reach a goal state? We answer these questions positively by proposing a multi-step procedure that first learns a world model that goes backward in time, secondly generates goal-reaching backward trajectories, thirdly improves those sequences using shortest path finding algorithms, and finally trains a neural network policy by imitation learning. We evaluate our method on a deterministic maze environment where the observations are 64× 64 pixel bird's eye images and can show that it consistently reaches several goals.