# Investigating practical linear temporal difference learning

arXiv: https://arxiv.org/abs/1602.08771 · HTML: https://mlsanity.com/paper/1602.08771/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2016.

## Built on by (most cited first, 13 of 13)

- [Empirical Design in Reinforcement Learning](https://mlsanity.com/paper/2304.01315/) (2023)
- [A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning](https://mlsanity.com/paper/2104.13844/) (2021)
- [Projection-Free Algorithm for Stochastic Bi-level Optimization](https://mlsanity.com/paper/2110.11721/) (2021)
- [Robust Losses for Learning Value Functions](https://mlsanity.com/paper/2205.08464/) (2022)
- [Revisiting a Design Choice in Gradient Temporal Difference Learning](https://mlsanity.com/paper/2308.01170/) (2023)
- [Deep Reinforcement Learning with Gradient Eligibility Traces](https://mlsanity.com/paper/2507.09087/) (2025)
- [An Empirical Comparison of Off-policy Prediction Learning Algorithms in the Four Rooms Environment](https://mlsanity.com/paper/2109.05110/) (2021)
- [An Empirical Comparison of Off-policy Prediction Learning Algorithms on the Collision Task](https://mlsanity.com/paper/2106.00922/) (2021)
- [A new Gradient TD Algorithm with only One Step-size: Convergence Rate Analysis using L-λ Smoothness](https://mlsanity.com/paper/2307.15892/) (2023)
- [The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning](https://mlsanity.com/paper/2407.18840/) (2024)
- [Importance Sampling Placement in Off-Policy Temporal-Difference Methods](https://mlsanity.com/paper/2203.10172/) (2022)
- [Gradient Descent Temporal Difference-difference Learning](https://mlsanity.com/paper/2209.04624/) (2022)
- [Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learning](https://mlsanity.com/paper/2602.12375/) (2026)

## Abstract (arXiv)

Off-policy reinforcement learning has many applications including: learning from demonstration, learning multiple goal seeking policies in parallel, and representing predictive knowledge. Recently there has been an proliferation of new policy-evaluation algorithms that fill a longstanding algorithmic void in reinforcement learning: combining robustness to off-policy sampling, function approximation, linear complexity, and temporal difference (TD) updates. This paper contains two main contributions. First, we derive two new hybrid TD policy-evaluation algorithms, which fill a gap in this collection of algorithms. Second, we perform an empirical comparison to elicit which of these new linear TD methods should be preferred in different situations, and make concrete suggestions about practical use.