mlsanityData to 7 Oct 2026

AWAC: Accelerating Online Reinforcement Learning with Offline Datasets

arXiv 2006.09359 ↗First posted 16 Jun 2020cs.LG cs.RO stat.MLcited by 647

Not on the map. Why?

Cited by 370 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reinforcement learning (RL) provides an appealing formalism for learning control policies from experience. However, the classic active formulation of RL necessitates a lengthy active exploration process for each behavior, making it difficult to apply in real-world settings such as robotic control. If we can instead allow RL algorithms to effectively use previously collected data to aid the online learning process, such applications could be made substantially more practical: the prior data would provide a starting point that mitigates challenges due to exploration and sample complexity, while the online training enables the agent to perfect the desired skill. Such prior data could either constitute expert demonstrations or sub-optimal prior data that illustrates potentially useful transitions. While a number of prior methods have either used optimal demonstrations to bootstrap RL, or have used sub-optimal data to train purely offline, it remains exceptionally difficult to train a policy with offline data and actually continue to improve it further with online RL. In this paper we analyze why this problem is so challenging, and propose an algorithm that combines sample efficient dynamic programming with maximum likelihood policy updates, providing a simple and effective framework that is able to leverage large amounts of offline data and then quickly perform online fine-tuning of RL policies. We show that our method, advantage weighted actor critic (AWAC), enables rapid learning of skills with a combination of prior demonstration data and online experience. We demonstrate these benefits on simulated and real-world robotics domains, including dexterous manipulation with a real multi-fingered hand, drawer opening with a robotic arm, and rotating a valve. Our results show that incorporating prior data can reduce the time required to learn a range of robotic skills to practical time-scales.

Key facts. Cited by 647 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 4; 2021: 50; 2022: 96; 2023: 130; 2024: 118; 2025: 122; 2026: 127.

647 papers.

Sort:

By area:

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  3. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  4. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  5. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  6. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  7. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  8. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  9. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  10. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  11. 2024Diffusion Policy Policy Optimizationcited by 255
  12. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
38 more
  1. 2023Pretraining Language Models with Human Preferencescited by 245
  2. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  3. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  4. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  5. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  6. 2022Online Decision Transformercited by 194
  7. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  8. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  9. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  10. 2025Flow Q-Learningcited by 162
  11. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  12. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  13. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  14. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  15. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  16. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
  17. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  18. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  19. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  20. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  21. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  22. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  23. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  24. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  25. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  26. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  27. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  28. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  29. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  30. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  31. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  32. 2022Jump-Start Reinforcement Learningcited by 102
  33. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  34. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  35. 2025Reinforcement Learning with Action Chunkingcited by 97
  36. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93
  37. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
  38. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 91

50 most cited of 647 All 647 (JSON)