mlsanityData to 7 Oct 2026

Temporal Difference Models: Model-Free Deep RL for Model-Based Control

arXiv 1802.09081 ↗First posted 25 Feb 2018cs.LGcited by 89

Not on the map. Why?

Abstract (arXiv)

Model-free reinforcement learning (RL) is a powerful, general tool for learning complex behaviors. However, its sample efficiency is often impractically large for solving challenging real-world problems, even with off-policy algorithms such as Q-learning. A limiting factor in classic model-free RL is that the learning signal consists only of scalar rewards, ignoring much of the rich information contained in state transition tuples. Model-based RL uses this information, by training a predictive model, but often does not achieve the same asymptotic performance as model-free RL due to model bias. We introduce temporal difference models (TDMs), a family of goal-conditioned value functions that can be trained with model-free learning and used for model-based control. TDMs combine the benefits of model-free and model-based RL: they leverage the rich information in state transitions to learn very efficiently, while still attaining asymptotic performance that exceeds that of direct model-based RL methods. Our experimental results show that, on a range of continuous control tasks, TDMs provide a substantial improvement in efficiency compared to state-of-the-art model-based and model-free methods.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 3; 2021: 26; 2022: 16; 2023: 20; 2024: 6; 2025: 11; 2026: 7.

89 papers.

Sort:

By area:

  1. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  2. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  3. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  4. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  5. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  6. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
  7. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  8. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  9. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52
  10. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learningcited by 42
  11. 2021Autonomous Reinforcement Learning via Subgoal Curriculacited by 32
  12. 2021Model-Based Reinforcement Learning via Latent-Space Collocationcited by 27
38 more
  1. 2021StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learningcited by 27
  2. 2021Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learningcited by 25
  3. 2021Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learningcited by 23
  4. 2021Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawingcited by 22
  5. 2022Context-Aware Language Modeling for Goal-Oriented Dialogue Systemscited by 22
  6. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representationscited by 20
  7. 2021Outcome-Driven Reinforcement Learning via Variational Inferencecited by 17
  8. 2021Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learningcited by 17
  9. 2021Understanding the World Through Actioncited by 17
  10. 2022Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learningcited by 16
  11. 2021Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learningcited by 15
  12. 2021Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learningcited by 15
  13. 2021Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learningcited by 15
  14. 2023Learning Video-Conditioned Policies for Unseen Manipulation Taskscited by 15
  15. 2024Knockoff-Guided Feature Selection via A Single Pre-trained Reinforced Agentcited by 14
  16. 2020Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learningcited by 12
  17. 2022A System for Morphology-Task Generalization via Unified Representation and Behavior Distillationcited by 12
  18. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 12
  19. 2021Braxlines: Fast and Interactive Toolkit for RL-driven Behavior Engineering beyond Reward Maximizationcited by 11
  20. 2022Generalised Policy Improvement with Geometric Policy Compositioncited by 11
  21. 2023H-GAP: Humanoid Control with a Generalist Plannercited by 11
  22. 2021Adjacency constraint for efficient hierarchical reinforcement learningcited by 10
  23. 2022Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learningcited by 10
  24. 2021DisCo RL: Distribution-Conditioned Reinforcement Learning for General-Purpose Policiescited by 9
  25. 2022Active Exploration for Robotic Manipulationcited by 9
  26. 2022PG3: Policy-Guided Planning for Generalized Policy Generationcited by 8
  27. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
  28. 2023PDSketch: Integrated Planning Domain Programming and Learningcited by 7
  29. 2024Model-based Reinforcement Learning for Parameterized Action Spacescited by 7
  30. 2022Integrating Symmetry into Differentiable Planning with Steerable Convolutionscited by 6
  31. 2022Scaling up and Stabilizing Differentiable Planning with Implicit Differentiationcited by 6
  32. 2023Online Reinforcement Learning in Non-Stationary Context-Driven Environmentscited by 6
  33. 2025Knowledge Retention for Continual Model-Based Reinforcement Learningcited by 6
  34. 2021Deep Reinforcement Learning for Conservation Decisionscited by 5
  35. 2022Equipping Black-Box Policies with Model-Based Advice for Stable Nonlinear Controlcited by 5
  36. 2023Curricular Subgoals for Inverse Reinforcement Learningcited by 5
  37. 2025Safe Multi-Agent Navigation guided by Goal-Conditioned Safe Reinforcement Learningcited by 5
  38. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 5

50 most cited of 89 All 89 (JSON)