mlsanityData to 7 Oct 2026

Don't Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learning

arXiv 2207.04703 ↗First posted 11 Jul 2022cs.RO cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) algorithms hold the promise of enabling autonomous skill acquisition for robotic systems. However, in practice, real-world robotic RL typically requires time consuming data collection and frequent human intervention to reset the environment. Moreover, robotic policies learned with RL often fail when deployed beyond the carefully controlled setting in which they were learned. In this work, we study how these challenges can all be tackled by effective utilization of diverse offline datasets collected from previously seen tasks. When faced with a new task, our system adapts previously learned skills to quickly learn to both perform the new task and return the environment to an initial state, effectively performing its own environment reset. Our empirical results demonstrate that incorporating prior data into robotic reinforcement learning enables autonomous learning, substantially improves sample-efficiency of learning, and enables better generalization. Project website: https://sites.google.com/view/ariel-berkeley/

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 54 of its 64 references that are in the map. First posted on arXiv 11 Jul 2022.

Builds on

54 of 64 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2015Continuous control with deep reinforcement learningcited by 3,279
  7. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  12. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
38 more
  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  4. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  8. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  9. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  10. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  11. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  12. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  13. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  14. 2020Critic Regularized Regressioncited by 269
  15. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  16. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  17. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  18. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  19. 2016What makes ImageNet good for transfer learning?cited by 192
  20. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 176
  21. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  22. 2018DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skillscited by 153
  23. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
  24. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 138
  25. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  26. 2020Learning to Walk in the Real World with Minimal Human Effortcited by 103
  27. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  28. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
  29. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  30. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
  31. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  32. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52
  33. 2021Learning to See before Learning to Act: Visual Pre-training for Manipulationcited by 50
  34. 2020Reset-Free Lifelong Learning with Skill-Space Planningcited by 38
  35. 2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
  36. 2021Autonomous Reinforcement Learning via Subgoal Curriculacited by 32
  37. 2020Continual Learning of Control Primitives: Skill Discovery via Reset-Gamescited by 29
  38. 2017Deep Q-learning from Demonstrationscited by 27

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 10; 2024: 6; 2025: 3; 2026: 5.

28 papers.

Sort:

By area:

  1. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  2. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  3. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 72
  4. 2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learningcited by 49
  5. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
  6. 2023Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learningcited by 29
  7. 2022Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Taskscited by 17
  8. 2026PlayWorld: Learning Robot World Models from Autonomous Playcited by 16
  9. 2022On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learningcited by 15
  10. 2024Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learningcited by 14
  11. 2023FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicingcited by 13
  12. 2023Instructed Diffuser with Temporal Condition Guidance for Offline Reinforcement Learningcited by 12
16 more
  1. 2023Replay across Experiments: A Natural Extension of Off-Policy RLcited by 10
  2. 2023Skill-Critic: Refining Learned Skills for Hierarchical Reinforcement Learningcited by 6
  3. 2023Hybrid Search for Efficient Planning with Completeness Guaranteescited by 6
  4. 2025SafeMimic: Towards Safe and Autonomous Human-to-Robot Imitation for Mobile Manipulationcited by 5
  5. 2026From Noise to Intent: Anchoring Generative VLA Policies with Residual Bridgescited by 3
  6. 2023Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculumcited by 2
  7. 2024Closure Discovery for Coarse-Grained Partial Differential Equations Using Grid-based Reinforcement Learningcited by 2
  8. 2024The State of Robot Motion Generationcited by 2
  9. 2026Scaling Sim-to-Real VLA Reinforcement Learning with Generative 3D Worldscited by 2
  10. 2023Modifying RL Policies with Imagined Actions: How Predictable Policies Can Enable Users to Perform Novel Taskscited by 1
  11. 2025Visuomotor Grasping with World Models for Surgical Robotscited by 1
  12. 2022Symmetry Detection in Trajectory Data for More Meaningful Reinforcement Learning Representationscited by 0
  13. 2023Learning Embeddings for Sequential Tasks Using Population of Agentscited by 0
  14. 2024Imagining In-distribution States: How Predictable Robot Behavior Can Enable User Control Over Learned Policiescited by 0
  15. 2026Expert Behavior Prior Reinforcement Learningcited by 0
  16. 2026Latent evolving World Action Modelcited by 0