# Recall Traces: Backtracking Models for Efficient Reinforcement Learning

arXiv: https://arxiv.org/abs/1804.00379 · HTML: https://mlsanity.com/paper/1804.00379/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2018.

## Built on by (most cited first, 23 of 23)

- [A Survey on Model-based Reinforcement Learning](https://mlsanity.com/paper/2206.09328/) (2022)
- [Retrieval-Augmented Reinforcement Learning](https://mlsanity.com/paper/2202.08417/) (2022)
- [Offline Reinforcement Learning with Reverse Model-based Imagination](https://mlsanity.com/paper/2110.00188/) (2021)
- [PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning](https://mlsanity.com/paper/2106.04152/) (2021)
- [Generative Adversarial Self-Imitation Learning](https://mlsanity.com/paper/1812.00950/) (2018)
- [Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination](https://mlsanity.com/paper/2206.07989/) (2022)
- [Topological Experience Replay](https://mlsanity.com/paper/2203.15845/) (2022)
- [Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy](https://mlsanity.com/paper/2207.12141/) (2022)
- [Generative predecessor models for sample-efficient imitation learning](https://mlsanity.com/paper/1904.01139/) (2019)
- [Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis](https://mlsanity.com/paper/2306.16803/) (2023)
- [Hindsight PRIORs for Reward Learning from Human Preferences](https://mlsanity.com/paper/2404.08828/) (2024)
- [Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets](https://mlsanity.com/paper/2406.01150/) (2024)
- [Offline Imitation Learning with Model-based Reverse Augmentation](https://mlsanity.com/paper/2406.12550/) (2024)
- [Robust Imitation of a Few Demonstrations with a Backwards Model](https://mlsanity.com/paper/2210.09337/) (2022)
- [Learning What To Do by Simulating the Past](https://mlsanity.com/paper/2104.03946/) (2021)
- [Solving Sokoban with forward-backward reinforcement learning](https://mlsanity.com/paper/2105.01904/) (2021)
- [Introspective Experience Replay: Look Back When Surprised](https://mlsanity.com/paper/2206.03171/) (2022)
- [Backward Imitation and Forward Reinforcement Learning via Bi-directional Model Rollouts](https://mlsanity.com/paper/2208.02434/) (2022)
- [Backward Learning for Goal-Conditioned Policies](https://mlsanity.com/paper/2312.05044/) (2023)
- [Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning](https://mlsanity.com/paper/2404.04682/) (2024)
- [Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning](https://mlsanity.com/paper/2405.03878/) (2024)
- [A Tighter Convergence Proof of Reverse Experience Replay](https://mlsanity.com/paper/2408.16999/) (2024)
- [Time Reversal Symmetry for Efficient Robotic Manipulations in Deep Reinforcement Learning](https://mlsanity.com/paper/2505.13925/) (2025)

## Abstract (arXiv)

In many environments only a tiny subset of all states yield high reward. In these cases, few of the interactions with the environment provide a relevant learning signal. Hence, we may want to preferentially train on those high-reward states and the probable trajectories leading to them. To this end, we advocate for the use of a backtracking model that predicts the preceding states that terminate at a given high-reward state. We can train a model which, starting from a high value state (or one that is estimated to have high value), predicts and sample for which the (state, action)-tuples may have led to that high value state. These traces of (state, action) pairs, which we refer to as Recall Traces, sampled from this backtracking model starting from a high value state, are informative as they terminate in good states, and hence we can use these traces to improve a policy. We provide a variational interpretation for this idea and a practical algorithm in which the backtracking model samples from an approximate posterior distribution over trajectories which lead to large rewards. Our method improves the sample efficiency of both on- and off-policy RL algorithms across several environments and tasks.