# Generalized Emphatic Temporal Difference Learning: Bias-Variance Analysis

arXiv: https://arxiv.org/abs/1509.05172 · HTML: https://mlsanity.com/paper/1509.05172/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2015.

## Built on by (most cited first, 17 of 17)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning](https://mlsanity.com/paper/2104.13844/) (2021)
- [Emphatic Algorithms for Deep Reinforcement Learning](https://mlsanity.com/paper/2106.11779/) (2021)
- [Truncated Emphatic Temporal Difference Methods for Prediction and Control](https://mlsanity.com/paper/2108.05338/) (2021)
- [LLMs Can Learn to Reason Via Off-Policy RL](https://mlsanity.com/paper/2602.19362/) (2026)
- [Revisiting a Design Choice in Gradient Temporal Difference Learning](https://mlsanity.com/paper/2308.01170/) (2023)
- [SOPE: Spectrum of Off-Policy Estimators](https://mlsanity.com/paper/2111.03936/) (2021)
- [Selective Credit Assignment](https://mlsanity.com/paper/2202.09699/) (2022)
- [Hindsight-DICE: Stable Credit Assignment for Deep Reinforcement Learning](https://mlsanity.com/paper/2307.11897/) (2023)
- [An Empirical Comparison of Off-policy Prediction Learning Algorithms in the Four Rooms Environment](https://mlsanity.com/paper/2109.05110/) (2021)
- [An Empirical Comparison of Off-policy Prediction Learning Algorithms on the Collision Task](https://mlsanity.com/paper/2106.00922/) (2021)
- [PER-ETD: A Polynomially Efficient Emphatic Temporal Difference Learning Method](https://mlsanity.com/paper/2110.06906/) (2021)
- [A new Gradient TD Algorithm with only One Step-size: Convergence Rate Analysis using L-λ Smoothness](https://mlsanity.com/paper/2307.15892/) (2023)
- [Analysis of Off-Policy n-Step TD-Learning with Linear Function Approximation](https://mlsanity.com/paper/2502.08941/) (2025)
- [Time After Time: Deep-Q Effect Estimation for Interventions on When and What to do](https://mlsanity.com/paper/2503.15890/) (2025)
- [Regularized Centered Emphatic Temporal Difference Learning](https://mlsanity.com/paper/2605.04100/) (2026)
- [Regularized Emphatic Temporal-Difference Learning: Stability under Constant Stepsizes](https://mlsanity.com/paper/2609.19170/) (2026)

## Abstract (arXiv)

We consider the off-policy evaluation problem in Markov decision processes with function approximation. We propose a generalization of the recently introduced emphatic temporal differences (ETD) algorithm citepSuttonMW15, which encompasses the original ETD(λ), as well as several other off-policy evaluation algorithms as special cases. We call this framework ETD, where our introduced parameter β controls the decay rate of an importance-sampling term. We study conditions under which the projected fixed-point equation underlying ETD involves a contraction operator, allowing us to present the first asymptotic error bounds (bias) for ETD. Our results show that the original ETD algorithm always involves a contraction operator, and its bias is bounded. Moreover, by controlling β, our proposed generalization allows trading-off bias for variance reduction, thereby achieving a lower total error.