Data to 14 Oct 2026

Efficient Online Reinforcement Learning with Offline Data

arXiv 2302.02948 ↗First posted 6 Feb 2023cs.LG cs.AIcited by 363

Not on the map. Why?

Cited by 184 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Sample efficiency and exploration remain major challenges in online reinforcement learning (RL). A powerful approach that can be applied to address these issues is the inclusion of offline data, such as prior trajectories from a human expert or a sub-optimal exploration policy. Previous methods have relied on extensive modifications and additional complexity to ensure the effective use of this data. Instead, we ask: can we simply apply existing off-policy methods to leverage offline data when learning online? In this work, we demonstrate that the answer is yes; however, a set of minimal but important changes to existing off-policy RL algorithms are required to achieve reliable performance. We extensively ablate these design choices, demonstrating the key factors that most affect performance, and arrive at a set of recommendations that practitioners can readily apply, whether their data comprise a small number of expert demonstrations or large volumes of sub-optimal trajectories. We see that correct application of these simple recommendations can provide a 2.5× improvement over existing approaches across a diverse set of competitive benchmarks, with no additional computational overhead. We have released our code at https://github.com/ikostrikov/rlpd.

Key facts. Cited by 363 papers in the map (data through 14 Oct 2026). Builds on 36 of its 55 references that are in the map. First posted on arXiv 6 Feb 2023.

Builds on

36 of 55 references.

References · 36

2022Training language models to follow instructions with human feedbackcited by 15,971
2016Layer Normalizationcited by 5,669
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
2015Continuous control with deep reinforcement learningcited by 3,407
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
2015Prioritized Experience Replaycited by 1,099

24 more

2017Deep Reinforcement Learning that Matterscited by 754
2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 62
2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
2021Offline Reinforcement Learning as Anti-Explorationcited by 44
2021AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scalecited by 44
2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
2022On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrationscited by 19
2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15
2021OffCon3: What is state of the art anyway?cited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 37; 2024: 74; 2025: 106; 2026: 146.

363 papers.

Sort:

By area:

2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
2024Diffusion Policy Policy Optimizationcited by 280
2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
2025Flow Q-Learningcited by 184
2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 176
2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 154
2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 144
2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 136
2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126

38 more

2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
2025Reinforcement Learning with Action Chunkingcited by 107
2023Synthetic Experience Replaycited by 101
2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 100
2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 90
2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 86
2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 79
2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 78
2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 70
2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 69
2023Imitation Bootstrapped Reinforcement Learningcited by 64
2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 63
2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 62
2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 60
2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 59
2024Normalization and effective learning rates in reinforcement learningcited by 58
2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 55
2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
2026RL Token: Bootstrapping Online RL with Vision-Language-Action Modelscited by 49
2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
2024RLDG: Robotic Generalist Policy Distillation via Reinforcement Learningcited by 47
2024Dataset Reset Policy Optimization for RLHFcited by 41
2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 38
2026ENPIRE: Agentic Robot Policy Self-Improvement in the Real Worldcited by 38
2026Q-learning with Adjoint Matchingcited by 37
2025World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulationcited by 35
2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 34
2024Hybrid Inverse Reinforcement Learningcited by 32
2023Cross-Domain Policy Adaptation via Value-Guided Data Filteringcited by 31
2024Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flightcited by 31
2023RLIF: Interactive Imitation Learning as Reinforcement Learningcited by 30
2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 29
2025Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Correctionscited by 29
2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 26
2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25

50 most cited of 363 All 363 (JSON)