# Emphatic Temporal-Difference Learning

arXiv: https://arxiv.org/abs/1507.01569 · HTML: https://mlsanity.com/paper/1507.01569/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jul 2015.

## Built on by (most cited first, 16 of 16)

- [DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization](https://mlsanity.com/paper/2112.04716/) (2021)
- [A Survey of Temporal Credit Assignment in Deep Reinforcement Learning](https://mlsanity.com/paper/2312.01072/) (2023)
- [Flexible Option Learning](https://mlsanity.com/paper/2112.03097/) (2021)
- [Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networks](https://mlsanity.com/paper/2209.07670/) (2022)
- [LLMs Can Learn to Reason Via Off-Policy RL](https://mlsanity.com/paper/2602.19362/) (2026)
- [Correcting discount-factor mismatch in on-policy policy gradient methods](https://mlsanity.com/paper/2306.13284/) (2023)
- [Revisiting a Design Choice in Gradient Temporal Difference Learning](https://mlsanity.com/paper/2308.01170/) (2023)
- [SOPE: Spectrum of Off-Policy Estimators](https://mlsanity.com/paper/2111.03936/) (2021)
- [Bellman Calibration for V-Learning in Offline Reinforcement Learning](https://mlsanity.com/paper/2512.23694/) (2025)
- [PER-ETD: A Polynomially Efficient Emphatic Temporal Difference Learning Method](https://mlsanity.com/paper/2110.06906/) (2021)
- [Off-Policy Evaluation for Action-Dependent Non-Stationary Environments](https://mlsanity.com/paper/2301.10330/) (2023)
- [Backstepping Temporal Difference Learning](https://mlsanity.com/paper/2302.09875/) (2023)
- [A new Gradient TD Algorithm with only One Step-size: Convergence Rate Analysis using L-λ Smoothness](https://mlsanity.com/paper/2307.15892/) (2023)
- [Can Active Sampling Reduce Causal Confusion in Offline Reinforcement Learning?](https://mlsanity.com/paper/2312.17168/) (2023)
- [Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting](https://mlsanity.com/paper/2512.23805/) (2025)
- [CurveRL: Principled Distribution-Aware Context Reweighting for LLM Reasoning](https://mlsanity.com/paper/2605.24331/) (2026)

## Abstract (arXiv)

Emphatic algorithms are temporal-difference learning algorithms that change their effective state distribution by selectively emphasizing and de-emphasizing their updates on different time steps. Recent works by Sutton, Mahmood and White (2015), and Yu (2015) show that by varying the emphasis in a particular way, these algorithms become stable and convergent under off-policy training with linear function approximation. This paper serves as a unified summary of the available results from both works. In addition, we demonstrate the empirical benefits from the flexibility of emphatic algorithms, including state-dependent discounting, state-dependent bootstrapping, and the user-specified allocation of function approximation resources.