# Generative Temporal Difference Learning for Infinite-Horizon Prediction

arXiv: https://arxiv.org/abs/2010.14496 · HTML: https://mlsanity.com/paper/2010.14496/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2020.

## Built on by (most cited first, 30 of 45)

- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Contrastive Difference Predictive Coding](https://mlsanity.com/paper/2310.20141/) (2023)
- [Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning](https://mlsanity.com/paper/2402.03570/) (2024)
- [Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objective](https://mlsanity.com/paper/2209.08466/) (2022)
- [Temporal Difference Flows](https://mlsanity.com/paper/2503.09817/) (2025)
- [floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL](https://mlsanity.com/paper/2509.06863/) (2025)
- [Learning Latent Dynamic Robust Representations for World Models](https://mlsanity.com/paper/2405.06263/) (2024)
- [RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictability](https://mlsanity.com/paper/2309.00082/) (2023)
- [Understanding the World Through Action](https://mlsanity.com/paper/2110.12543/) (2021)
- [A Distributional Analogue to the Successor Representation](https://mlsanity.com/paper/2402.08530/) (2024)
- [Contrastive Value Learning: Implicit Models for Simple Offline RL](https://mlsanity.com/paper/2211.02100/) (2022)
- [Value Flows](https://mlsanity.com/paper/2510.07650/) (2025)
- [Prioritized Generative Replay](https://mlsanity.com/paper/2410.18082/) (2024)
- [Self-Supervised Reinforcement Learning that Transfers using Random Features](https://mlsanity.com/paper/2305.17250/) (2023)
- [Bootstrapped Representations in Reinforcement Learning](https://mlsanity.com/paper/2306.10171/) (2023)
- [Value function estimation using conditional diffusion models for control](https://mlsanity.com/paper/2306.07290/) (2023)
- [A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2310.06253/) (2023)
- [Intention-Conditioned Flow Occupancy Models](https://mlsanity.com/paper/2506.08902/) (2025)
- [Distributional Successor Features Enable Zero-Shot Policy Optimization](https://mlsanity.com/paper/2403.06328/) (2024)
- [Accelerating exploration and representation learning with offline pre-training](https://mlsanity.com/paper/2304.00046/) (2023)
- [Towards Robust Zero-Shot Reinforcement Learning](https://mlsanity.com/paper/2510.15382/) (2025)
- [Compositional Planning with Jumpy World Models](https://mlsanity.com/paper/2602.19634/) (2026)
- [Successor Feature Representations](https://mlsanity.com/paper/2110.15701/) (2021)
- [Mingling Foresight with Imagination: Model-Based Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2204.09418/) (2022)
- [Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2303.03955/) (2023)
- [Bellman Diffusion Models](https://mlsanity.com/paper/2407.12163/) (2024)
- [Taylor Expansion of Discount Factors](https://mlsanity.com/paper/2106.06170/) (2021)
- [Video Occupancy Models](https://mlsanity.com/paper/2407.09533/) (2024)

## Abstract (arXiv)

We introduce the γ-model, a predictive model of environment dynamics with an infinite probabilistic horizon. Replacing standard single-step models with γ-models leads to generalizations of the procedures central to model-based control, including the model rollout and model-based value estimation. The γ-model, trained with a generative reinterpretation of temporal difference learning, is a natural continuous analogue of the successor representation and a hybrid between model-free and model-based mechanisms. Like a value function, it contains information about the long-term future; like a standard predictive model, it is independent of task reward. We instantiate the γ-model as both a generative adversarial network and normalizing flow, discuss how its training reflects an inescapable tradeoff between training-time and testing-time compounding errors, and empirically investigate its utility for prediction and control.