# Temporal-Difference Networks

arXiv: https://arxiv.org/abs/1504.05539 · HTML: https://mlsanity.com/paper/1504.05539/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2015.

## Built on by (most cited first, 13 of 13)

- [Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directions](https://mlsanity.com/paper/2112.11561/) (2021)
- [A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation](https://mlsanity.com/paper/2106.06854/) (2021)
- [Exploring through Random Curiosity with General Value Functions](https://mlsanity.com/paper/2211.10282/) (2022)
- [Towards Safe, Explainable, and Regulated Autonomous Driving](https://mlsanity.com/paper/2111.10518/) (2021)
- [Learning State Representations from Random Deep Action-conditional Predictions](https://mlsanity.com/paper/2102.04897/) (2021)
- [Learning how to Interact with a Complex Interface using Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2204.10374/) (2022)
- [Learning Agent State Online with Recurrent Generate-and-Test](https://mlsanity.com/paper/2112.15236/) (2021)
- [A Decade of Deep Learning for Remote Sensing Spatiotemporal Fusion: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2504.00901/) (2025)
- [Predictive Representation Learning for Language Modeling](https://mlsanity.com/paper/2105.14214/) (2021)
- [A Unified Off-Policy Evaluation Approach for General Value Function](https://mlsanity.com/paper/2107.02711/) (2021)
- [Representing Knowledge as Predictions (and State as Knowledge)](https://mlsanity.com/paper/2112.06336/) (2021)
- [Investigating Action Encodings in Recurrent Neural Networks in Reinforcement Learning](https://mlsanity.com/paper/2605.16318/) (2026)
- [Predictive Representations for Skill Transfer in Reinforcement Learning](https://mlsanity.com/paper/2604.07016/) (2026)

## Abstract (arXiv)

We introduce a generalization of temporal-difference (TD) learning to networks of interrelated predictions. Rather than relating a single prediction to itself at a later time, as in conventional TD methods, a TD network relates each prediction in a set of predictions to other predictions in the set at a later time. TD networks can represent and apply TD learning to a much wider class of predictions than has previously been possible. Using a random-walk example, we show that these networks can be used to learn to predict by a fixed interval, which is not possible with conventional TD methods. Secondly, we show that if the inter-predictive relationships are made conditional on action, then the usual learning-efficiency advantage of TD methods over Monte Carlo (supervised learning) methods becomes particularly pronounced. Thirdly, we demonstrate that TD networks can learn predictive state representations that enable exact solution of a non-Markov problem. A very broad range of inter-predictive temporal relationships can be expressed in these networks. Overall we argue that TD networks represent a substantial extension of the abilities of TD methods and bring us closer to the goal of representing world knowledge in entirely predictive, grounded terms.