mlsanityData to 7 Oct 2026

Consistent On-Line Off-Policy Evaluation

arXiv 1702.07121 ↗First posted 23 Feb 2017stat.ML cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

The problem of on-line off-policy evaluation (OPE) has been actively studied in the last decade due to its importance both as a stand-alone problem and as a module in a policy improvement scheme. However, most Temporal Difference (TD) based solutions ignore the discrepancy between the stationary distribution of the behavior and target policies and its effect on the convergence limit when function approximation is applied. In this paper we propose the Consistent Off-Policy Temporal Difference (COP-TD(λ, β)) algorithm that addresses this issue and reduces this bias at some computational expense. We show that COP-TD(λ, β) can be designed to converge to the same value that would have been obtained by using on-policy TD(λ) with the target policy. Subsequently, the proposed scheme leads to a related and promising heuristic we call log-COP-TD(λ, β). Both algorithms have favorable empirical results to the current state of the art on-line OPE algorithms. Finally, our formulation sheds some new light on the recently proposed Emphatic TD learning.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Feb 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 4; 2021: 15; 2022: 9; 2023: 9; 2024: 7; 2025: 3; 2026: 4.

51 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  5. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  6. 2021Nearly Horizon-Free Offline Reinforcement Learningcited by 36
  7. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 30
  8. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  9. 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
  10. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  11. 2020Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficientcited by 24
  12. 2021Flexible Option Learningcited by 24
38 more
  1. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  2. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  3. 2021Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatchcited by 17
  4. 2023Reinforcement Learning in Low-Rank MDPs with Density Featurescited by 17
  5. 2022Grounding Aleatoric Uncertainty for Unsupervised Environment Designcited by 16
  6. 2021Continual Learning In Environments With Polynomial Mixing Timescited by 13
  7. 2023Hallucinated Adversarial Control for Conservative Offline Policy Evaluationcited by 13
  8. 2024A CMDP-within-online framework for Meta-Safe Reinforcement Learningcited by 12
  9. 2021Truncated Emphatic Temporal Difference Methods for Prediction and Controlcited by 10
  10. 2024Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximationcited by 10
  11. 2022Continual Auxiliary Task Learningcited by 9
  12. 2023Correcting discount-factor mismatch in on-policy policy gradient methodscited by 9
  13. 2022Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstractioncited by 7
  14. 2023Revisiting a Design Choice in Gradient Temporal Difference Learningcited by 7
  15. 2024On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learningcited by 6
  16. 2024Solving Continual Offline RL through Selective Weights Activation on Aligned Spacescited by 6
  17. 2021Learning Expected Emphatic Traces for Deep RLcited by 5
  18. 2021SOPE: Spectrum of Off-Policy Estimatorscited by 5
  19. 2021Off-Policy Actor-Critic with Emphatic Weightingscited by 5
  20. 2023Hindsight-DICE: Stable Credit Assignment for Deep Reinforcement Learningcited by 4
  21. 2022Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learningcited by 3
  22. 2022Causal Deep Reinforcement Learning Using Observational Datacited by 3
  23. 2022Behavior Estimation from Multi-Source Data for Offline Reinforcement Learningcited by 2
  24. 2023A Reinforcement Learning Framework for Dynamic Mediation Analysiscited by 2
  25. 2023A new Gradient TD Algorithm with only One Step-size: Convergence Rate Analysis using L-λ Smoothnesscited by 2
  26. 2024When Do Off-Policy and On-Policy Policy Gradient Methods Align?cited by 2
  27. 2025Average-DICE: Stationary Distribution Correction by Regressioncited by 2
  28. 2021A Unified Off-Policy Evaluation Approach for General Value Functioncited by 1
  29. 2022Importance Sampling Placement in Off-Policy Temporal-Difference Methodscited by 1
  30. 2023Projected Off-Policy Q-Learning (POP-QL) for Stabilizing Offline Reinforcement Learningcited by 1
  31. 2025Double Horizon Model-Based Policy Optimizationcited by 1
  32. 2025Fitted Q-Evaluation without Bellman Completeness via Occupancy Weightingcited by 1
  33. 2021Forward and Backward Bellman equations improve the efficiency of EM algorithm for DEC-POMDPcited by 0
  34. 2024Towards Provable Log Density Policy Gradientcited by 0
  35. 2024SimuDICE: Offline Policy Optimization Through World Model Updates and DICE Estimationcited by 0
  36. 2026Beyond Bellman: High-Order Generator Regression for Continuous-Time Policy Evaluationcited by 0
  37. 2026Behavior-Aware Auxiliary Corrections for Off-Policy Temporal-Difference Predictioncited by 0
  38. 2026DT2: Decision-Targeted Digital Twinscited by 0

50 most cited of 51 All 51 (JSON)