mlsanityData to 7 Oct 2026

An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning

arXiv 1503.04269 ↗First posted 14 Mar 2015cs.LGcited by 103

Not on the map. Why?

Cited by 21 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper we introduce the idea of improving the performance of parametric temporal-difference (TD) learning algorithms by selectively emphasizing or de-emphasizing their updates on different time steps. In particular, we show that varying the emphasis of linear TD(λ)'s updates in a particular way causes its expected update to become stable under off-policy training. The only prior model-free TD methods to achieve this with per-step computation linear in the number of function approximation parameters are the gradient-TD family of methods including TDC, GTD(λ), and GQ(λ). Compared to these methods, our emphatic TD(λ)_ is simpler and easier to use; it has only one learned parameter vector and one step-size parameter. Our treatment includes general state-dependent discounting and bootstrapping functions, and a way of specifying varying degrees of interest in accurately valuing different states.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Mar 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 30; 2022: 20; 2023: 17; 2024: 16; 2025: 10; 2026: 8.

103 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  3. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  4. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  5. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  6. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  7. 2025WorldGym: World Model as An Environment for Policy Evaluationcited by 61
  8. 2023Empirical Design in Reinforcement Learningcited by 54
  9. 2021Breaking the Deadly Triad with a Target Networkcited by 47
  10. 2021Offline Reinforcement Learning with Reverse Model-based Imaginationcited by 47
  11. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 30
  12. 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
38 more
  1. 2021Active Offline Policy Selectioncited by 26
  2. 2021Online Bootstrap Inference For Policy Evaluation in Reinforcement Learningcited by 26
  3. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  4. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  5. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximationcited by 22
  6. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
  7. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  8. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  9. 2021Doubly Robust Off-Policy Actor-Critic: Convergence and Optimalitycited by 20
  10. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  11. 2021Finite-Sample Analysis of Off-Policy TD-Learning via Generalized Bellman Operatorscited by 17
  12. 2020DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPscited by 16
  13. 2022Regularized Q-learningcited by 16
  14. 2022Grounding Aleatoric Uncertainty for Unsupervised Environment Designcited by 16
  15. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  16. 2021Reinforcement Learning Under Algorithmic Triagecited by 15
  17. 2023TD Convergence: An Optimization Perspectivecited by 15
  18. 2025Rethinking the Foundations for Continual Reinforcement Learningcited by 15
  19. 2022Uncovering Instabilities in Variational-Quantum Deep Q-Networkscited by 14
  20. 2023Hallucinated Adversarial Control for Conservative Offline Policy Evaluationcited by 13
  21. 2021Emphatic Algorithms for Deep Reinforcement Learningcited by 12
  22. 2022Goal-Space Planning with Subgoal Modelscited by 12
  23. 2023Efficient Policy Evaluation with Offline Data Informed Behavior Policy Designcited by 12
  24. 2023Bootstrapped Representations in Reinforcement Learningcited by 11
  25. 2021Truncated Emphatic Temporal Difference Methods for Prediction and Controlcited by 10
  26. 2022Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlookcited by 10
  27. 2023Improving Offline RL by Blending Heuristicscited by 10
  28. 2024Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximationcited by 10
  29. 2022Continual Auxiliary Task Learningcited by 9
  30. 2024Efficient Multi-Policy Evaluation for Reinforcement Learningcited by 8
  31. 2021Bridging the Gap Between Target Networks and Functional Regularizationcited by 7
  32. 2023Revisiting a Design Choice in Gradient Temporal Difference Learningcited by 7
  33. 2024Doubly Optimal Policy Evaluation for Reinforcement Learningcited by 7
  34. 2021Preferential Temporal Difference Learningcited by 6
  35. 2024Grounded Answers for Multi-agent Decision-making Problem through Generative World Modelcited by 6
  36. 2025Relative Entropy Pathwise Policy Optimizationcited by 6
  37. 2021Learning Expected Emphatic Traces for Deep RLcited by 5
  38. 2021Off-Policy Actor-Critic with Emphatic Weightingscited by 5

50 most cited of 103 All 103 (JSON)