Data to 7 Oct 2026

Safe and Efficient Off-Policy Reinforcement Learning

arXiv 1606.02647 ↗First posted 8 Jun 2016cs.LG cs.AI stat.MLcited by 260

Not on the map. Why?

Cited by 22 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we take a fresh look at some old and new algorithms for off-policy, return-based reinforcement learning. Expressing these in a common form, we derive a novel algorithm, Retrace(λ), with three desired properties: (1) it has low variance; (2) it safely uses samples collected from any behaviour policy, whatever its degree of "off-policyness"; and (3) it is efficient as it makes the best use of samples collected from near on-policy behaviour policies. We analyze the contractive nature of the related operator under both off-policy policy evaluation and control settings and derive online sample-based algorithms. We believe this is the first return-based off-policy control algorithm converging a.s. to Q^* without the GLIE assumption (Greedy in the Limit with Infinite Exploration). As a corollary, we prove the convergence of Watkins' Q(λ), which was an open problem since 1989. We illustrate the benefits of Retrace(λ) on a standard suite of Atari 2600 games.

Key facts. Cited by 260 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 6; 2021: 63; 2022: 44; 2023: 30; 2024: 31; 2025: 36; 2026: 48.

260 papers.

Sort:

By area:

2020MOPO: Model-based Offline Policy Optimizationcited by 630
2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
2021Offline RL Without Off-Policy Evaluationcited by 150
2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
2018Self-Imitation Learningcited by 135
2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
2021Benchmarks for Deep Off-Policy Evaluationcited by 89
2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87

38 more

2021Mitigating Political Bias in Language Models Through Reinforced Calibrationcited by 73
2024Simplifying Deep Temporal Difference Learningcited by 71
2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 70
2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agentscited by 68
2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
2025WorldGym: World Model as An Environment for Policy Evaluationcited by 61
2021A Lyapunov Theory for Finite-Sample Guarantees of Asynchronous Q-Learning and TD-Learning Variantscited by 57
2021Muesli: Combining Improvements in Policy Optimizationcited by 53
2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 53
2021Scientific multi-agent reinforcement learning for wall-models of turbulent flowscited by 52
2021Adversarially Guided Actor-Criticcited by 47
2023Stochastic Optimal Control Matchingcited by 39
2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
2021Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policiescited by 37
2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 37
2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Taskscited by 32
2026Your Group-Relative Advantage Is Biasedcited by 32
2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29
2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generationcited by 29
2022Human-level Atari 200x fastercited by 28
2025Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystemcited by 28
2023Second Thoughts are Best: Learning to Re-Align With Human Values from Text Editscited by 27
2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithmcited by 26
2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
2021Active Offline Policy Selectioncited by 26
2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
2021Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Explorationcited by 25
2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximationcited by 22
2021Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learningcited by 21
2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
2022General Intelligence Requires Rethinking Explorationcited by 20
2021Revisiting Peng's Q(λ) for Modern Reinforcement Learningcited by 19
2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
2021Discovering Diverse Solutions in Deep Reinforcement Learning by Maximizing State-Action-Based Mutual Informationcited by 18
2024Transcendence: Generative Models Can Outperform The Experts That Train Themcited by 18
2021A Survey on Deep Reinforcement Learning for Audio-Based Applicationscited by 17

50 most cited of 260 All 260 (JSON)