mlsanityData to 14 Oct 2026

Adaptive Trade-Offs in Off-Policy Learning

arXiv 1910.07478 ↗First posted 16 Oct 2019cs.LG stat.MLcited by 20

Not on the map. Why?

Abstract (arXiv)

A great variety of off-policy learning algorithms exist in the literature, and new breakthroughs in this area continue to be made, improving theoretical understanding and yielding state-of-the-art reinforcement learning algorithms. In this paper, we take a unifying view of this space of algorithms, and consider their trade-offs of three fundamental quantities: update variance, fixed-point bias, and contraction rate. This leads to new perspectives of existing methods, and also naturally yields novel algorithms for off-policy evaluation and control. We develop one such algorithm, C-trace, demonstrating that it is able to more efficiently make these trade-offs than existing methods in use, and that it can be scaled to yield state-of-the-art performance in large-scale environments.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 6; 2022: 5; 2023: 3; 2024: 3; 2025: 0; 2026: 2.

20 papers.

Sort:

By area:

  1. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  2. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  3. 2021Revisiting Peng's Q(λ) for Modern Reinforcement Learningcited by 19
  4. 2022Grounding Aleatoric Uncertainty for Unsupervised Environment Designcited by 16
  5. 2021On component interactions in two-stage recommender systemscited by 14
  6. 2023The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimationcited by 14
  7. 2020Self-Imitation Learning via Generalized Lower Bound Q-learningcited by 13
  8. 2022The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learningcited by 10
  9. 2021Measuring Progress in Deep Reinforcement Learning Sample Efficiencycited by 8
  10. 2024Skill or Luck? Return Decomposition via Advantage Functionscited by 8
  11. 2023Trajectory-Aware Eligibility Traces for Off-Policy Reinforcement Learningcited by 6
  12. 2021Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluationcited by 5
8 more
  1. 2023Learning Curricula in Open-Ended Worldscited by 2
  2. 2024Highway Reinforcement Learningcited by 2
  3. 2022Safe-FinRL: A Low Bias and Variance Deep Reinforcement Learning Implementation for High-Freq Stock Tradingcited by 1
  4. 2024Off-policy Distributional Q(λ): Distributional RL without Importance Samplingcited by 1
  5. 2021Improving the Efficiency of Off-Policy Reinforcement Learning by Accounting for Past Decisionscited by 0
  6. 2022Marginalized Operators for Off-policy Reinforcement Learningcited by 0
  7. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learningcited by 0
  8. 2026COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learningcited by 0