mlsanityData to 7 Oct 2026

Generative Temporal Difference Learning for Infinite-Horizon Prediction

arXiv 2010.14496 ↗First posted 27 Oct 2020cs.LG cs.AIcited by 43

Not on the map. Why?

Abstract (arXiv)

We introduce the γ-model, a predictive model of environment dynamics with an infinite probabilistic horizon. Replacing standard single-step models with γ-models leads to generalizations of the procedures central to model-based control, including the model rollout and model-based value estimation. The γ-model, trained with a generative reinterpretation of temporal difference learning, is a natural continuous analogue of the successor representation and a hybrid between model-free and model-based mechanisms. Like a value function, it contains information about the long-term future; like a standard predictive model, it is independent of task reward. We instantiate the γ-model as both a generative adversarial network and normalizing flow, discuss how its training reflects an inescapable tradeoff between training-time and testing-time compounding errors, and empirically investigate its utility for prediction and control.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 5; 2023: 9; 2024: 12; 2025: 6; 2026: 7.

43 papers.

Sort:

By area:

  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  2. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  3. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  4. 2023Contrastive Difference Predictive Codingcited by 35
  5. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  6. 2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
  7. 2024Learning Latent Dynamic Robust Representations for World Modelscited by 20
  8. 2025Temporal Difference Flowscited by 20
  9. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  10. 2021Understanding the World Through Actioncited by 17
  11. 2023RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictabilitycited by 17
  12. 2022Contrastive Value Learning: Implicit Models for Simple Offline RLcited by 15
31 more
  1. 2024A Distributional Analogue to the Successor Representationcited by 15
  2. 2024Prioritized Generative Replaycited by 13
  3. 2025Value Flowscited by 12
  4. 2023Bootstrapped Representations in Reinforcement Learningcited by 11
  5. 2023Self-Supervised Reinforcement Learning that Transfers using Random Featurescited by 10
  6. 2023Value function estimation using conditional diffusion models for controlcited by 10
  7. 2023A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learningcited by 9
  8. 2024Distributional Successor Features Enable Zero-Shot Policy Optimizationcited by 8
  9. 2023Accelerating exploration and representation learning with offline pre-trainingcited by 7
  10. 2025Intention-Conditioned Flow Occupancy Modelscited by 7
  11. 2025Towards Robust Zero-Shot Reinforcement Learningcited by 7
  12. 2021Successor Feature Representationscited by 6
  13. 2022Mingling Foresight with Imagination: Model-Based Cooperative Multi-Agent Reinforcement Learningcited by 6
  14. 2023Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learningcited by 5
  15. 2021Taylor Expansion of Discount Factorscited by 4
  16. 2024Video Occupancy Modelscited by 4
  17. 2024Bellman Diffusion Modelscited by 4
  18. 2026Compositional Planning with Jumpy World Modelscited by 4
  19. 2024State Chrono Representation for Enhancing Generalization in Reinforcement Learningcited by 3
  20. 2024Off-Policy Maximum Entropy RL with Future State and Action Visitation Measurescited by 3
  21. 2024Associative memory inspires improvements for in-context learning using a novel attention residual stream architecturecited by 3
  22. 2024Exploration by Learning Diverse Skills through Successor State Measurescited by 2
  23. 2023Deep Generative Models for Decision-Making and Controlcited by 1
  24. 2026Can We Really Learn One Representation to Optimize All Rewards?cited by 1
  25. 2026Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  26. 2026Offline Reinforcement Learning with Universal Horizon Modelscited by 1
  27. 2026Temporal Difference Learning for Diffusion Modelscited by 1
  28. 2024Diminishing Return of Value Expansion Methodscited by 0
  29. 2025A Unified Framework for Zero-Shot Reinforcement Learningcited by 0
  30. 2026Maximum-Entropy Exploration with Future State-Action Visitation Measurescited by 0
  31. 2026Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learningcited by 0