mlsanityData to 14 Oct 2026

Q(λ) with Off-Policy Corrections

arXiv 1602.04951 ↗First posted 16 Feb 2016cs.AI cs.LG stat.MLcited by 48

Not on the map. Why?

Abstract (arXiv)

We propose and analyze an alternate approach to off-policy multi-step temporal difference learning, in which off-policy returns are corrected with the current Q-function in terms of rewards, rather than with the target policy in terms of transition probabilities. We prove that such approximate corrections are sufficient for off-policy convergence both in policy evaluation and control, provided certain conditions. These conditions relate the distance between the target and behavior policies, the eligibility trace parameter and the discount factor, and formalize an underlying tradeoff in off-policy TD(λ). We illustrate this theoretical relationship empirically on a continuous-state control task.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 Feb 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 11; 2022: 5; 2023: 13; 2024: 7; 2025: 4; 2026: 7.

48 papers.

Sort:

By area:

  1. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 79
  2. 2021A Lyapunov Theory for Finite-Sample Guarantees of Asynchronous Q-Learning and TD-Learning Variantscited by 57
  3. 2023Concentration of Contractive Stochastic Approximation: Additive and Multiplicative Noisecited by 26
  4. 2021Flexible Option Learningcited by 25
  5. 2023Order Matters: Agent-by-agent Policy Optimizationcited by 24
  6. 2021Revisiting Peng's Q(λ) for Modern Reinforcement Learningcited by 19
  7. 2021Finite-Sample Analysis of Off-Policy TD-Learning via Generalized Bellman Operatorscited by 17
  8. 2023Efficient Policy Evaluation with Offline Data Informed Behavior Policy Designcited by 14
  9. 2020Self-Imitation Learning via Generalized Lower Bound Q-learningcited by 13
  10. 2022The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learningcited by 10
  11. 2021On-Policy Model Errors in Reinforcement Learningcited by 6
  12. 2023Trajectory-Aware Eligibility Traces for Off-Policy Reinforcement Learningcited by 6
36 more
  1. 2023Sequential Counterfactual Risk Minimizationcited by 6
  2. 2023Value-aware Importance Weighting for Off-policy Reinforcement Learningcited by 4
  3. 2024Demystifying the Recency Heuristic in Temporal-Difference Learningcited by 4
  4. 2025PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Datacited by 4
  5. 2021Supervised Off-Policy Rankingcited by 3
  6. 2022Mitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approachcited by 3
  7. 2022An Approximate Policy Iteration Viewpoint of Actor-Critic Algorithmscited by 3
  8. 2023Probabilistic Offline Policy Ranking with Approximate Bayesian Computationcited by 3
  9. 2024Learning Uncertainty-Aware Temporally-Extended Actionscited by 3
  10. 2024CANDOR: Counterfactual ANnotated DOubly Robust Off-Policy Evaluationcited by 3
  11. 2023Off-Policy Evaluation for Action-Dependent Non-Stationary Environmentscited by 2
  12. 2023Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learningcited by 2
  13. 2024Highway Reinforcement Learningcited by 2
  14. 2025KL-Regularised Q-Learning: A Token-level Action-Value perspective on Online RLHFcited by 2
  15. 2025Learning to Undo: Rollback-Augmented Reinforcement Learning with Reversibility Signalscited by 2
  16. 2026Convergence of Monte Carlo Optimistic Policy Iteration: Beyond Uniform State-Action Updatescited by 2
  17. 2021Semi-On-Policy Training for Sample Efficient Multi-Agent Policy Gradientscited by 1
  18. 2023Never Explore Repeatedly in Multi-Agent Reinforcement Learningcited by 1
  19. 2024Off-policy Distributional Q(λ): Distributional RL without Importance Samplingcited by 1
  20. 2024Two-Step Q-Learningcited by 1
  21. 2026Almost Sure Convergence Rates of Stochastic Approximation and Reinforcement Learning via a Poisson-Moreau Driftcited by 1
  22. 2026Non-Asymptotic Convergence of Stochastic Iterative Algorithms: A Lyapunov Frameworkcited by 1
  23. 2021Learning Memory-Dependent Continuous Control from Demonstrationscited by 0
  24. 2021Greedy-Step Off-Policy Reinforcement Learningcited by 0
  25. 2021Improving the Efficiency of Off-Policy Reinforcement Learning by Accounting for Past Decisionscited by 0
  26. 2022Marginalized Operators for Off-policy Reinforcement Learningcited by 0
  27. 2022Knowing the Past to Predict the Future: Reinforcement Virtual Learningcited by 0
  28. 2023Distill Knowledge in Multi-task Reinforcement Learning with Optimal-Transport Regularizationcited by 0
  29. 2023Bias Resilient Multi-Step Off-Policy Goal-Conditioned Reinforcement Learningcited by 0
  30. 2023Conservative Exploration for Policy Optimization via Off-Policy Policy Evaluationcited by 0
  31. 2024A Multi-Step Minimax Q-learning Algorithm for Two-Player Zero-Sum Markov Gamescited by 0
  32. 2025Time After Time: Deep-Q Effect Estimation for Interventions on When and What to docited by 0
  33. 2026Adaptive TD-Lambda for Cooperative Multi-agent Reinforcement Learningcited by 0
  34. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learningcited by 0
  35. 2026Gated Q-learning: Add Off-Policy Bias to Tastecited by 0
  36. 2026Limiting-Kernel Q(λ): Bridging Short and Long Horizonscited by 0