# Adaptive Trade-Offs in Off-Policy Learning

arXiv: https://arxiv.org/abs/1910.07478 · HTML: https://mlsanity.com/paper/1910.07478/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2019.

## Built on by (most cited first, 20 of 20)

- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Offline Reinforcement Learning with Value-based Episodic Memory](https://mlsanity.com/paper/2110.09796/) (2021)
- [Revisiting Peng's Q(λ) for Modern Reinforcement Learning](https://mlsanity.com/paper/2103.00107/) (2021)
- [Grounding Aleatoric Uncertainty for Unsupervised Environment Design](https://mlsanity.com/paper/2207.05219/) (2022)
- [On component interactions in two-stage recommender systems](https://mlsanity.com/paper/2106.14979/) (2021)
- [The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation](https://mlsanity.com/paper/2305.18388/) (2023)
- [Self-Imitation Learning via Generalized Lower Bound Q-learning](https://mlsanity.com/paper/2006.07442/) (2020)
- [The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learning](https://mlsanity.com/paper/2207.07570/) (2022)
- [Measuring Progress in Deep Reinforcement Learning Sample Efficiency](https://mlsanity.com/paper/2102.04881/) (2021)
- [Skill or Luck? Return Decomposition via Advantage Functions](https://mlsanity.com/paper/2402.12874/) (2024)
- [Trajectory-Aware Eligibility Traces for Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2301.11321/) (2023)
- [Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation](https://mlsanity.com/paper/2106.13125/) (2021)
- [Learning Curricula in Open-Ended Worlds](https://mlsanity.com/paper/2312.03126/) (2023)
- [Highway Reinforcement Learning](https://mlsanity.com/paper/2405.18289/) (2024)
- [Safe-FinRL: A Low Bias and Variance Deep Reinforcement Learning Implementation for High-Freq Stock Trading](https://mlsanity.com/paper/2206.05910/) (2022)
- [Off-policy Distributional Q(λ): Distributional RL without Importance Sampling](https://mlsanity.com/paper/2402.05766/) (2024)
- [Improving the Efficiency of Off-Policy Reinforcement Learning by Accounting for Past Decisions](https://mlsanity.com/paper/2112.12281/) (2021)
- [Marginalized Operators for Off-policy Reinforcement Learning](https://mlsanity.com/paper/2203.16177/) (2022)
- [Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning](https://mlsanity.com/paper/2605.14779/) (2026)
- [COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning](https://mlsanity.com/paper/2610.09597/) (2026)

## Abstract (arXiv)

A great variety of off-policy learning algorithms exist in the literature, and new breakthroughs in this area continue to be made, improving theoretical understanding and yielding state-of-the-art reinforcement learning algorithms. In this paper, we take a unifying view of this space of algorithms, and consider their trade-offs of three fundamental quantities: update variance, fixed-point bias, and contraction rate. This leads to new perspectives of existing methods, and also naturally yields novel algorithms for off-policy evaluation and control. We develop one such algorithm, C-trace, demonstrating that it is able to more efficiently make these trade-offs than existing methods in use, and that it can be scaled to yield state-of-the-art performance in large-scale environments.