mlsanityData to 7 Oct 2026

Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trials

arXiv 2210.05178 ↗First posted 11 Oct 2022cs.RO cs.LGcited by 78

On the map: Offline RL and imitation learning from data → Real-world robot offline RL

Abstract (arXiv)

Progress in deep learning highlights the tremendous potential of utilizing diverse robotic datasets for attaining effective generalization and makes it enticing to consider leveraging broad datasets for attaining robust generalization in robotic learning as well. However, in practice, we often want to learn a new skill in a new environment that is unlikely to be contained in the prior data. Therefore we ask: how can we leverage existing diverse offline datasets in combination with small amounts of task-specific data to solve new tasks, while still enjoying the generalization benefits of training on large amounts of data? In this paper, we demonstrate that end-to-end offline RL can be an effective approach for doing this, without the need for any representation learning or vision-based pre-training. We present pre-training for robots (PTR), a framework based on offline RL that attempts to effectively learn new tasks by combining pre-training on existing robotic datasets with rapid fine-tuning on a new task, with as few as 10 demonstrations. PTR utilizes an existing offline RL method, conservative Q-learning (CQL), but extends it to include several crucial design decisions that enable PTR to actually work and outperform a variety of prior methods. To our knowledge, PTR is the first RL method that succeeds at learning new tasks in a new domain on a real WidowX robot with as few as 10 task demonstrations, by effectively leveraging an existing dataset of diverse multi-task robot data collected in a variety of toy kitchens. We also demonstrate that PTR can enable effective autonomous fine-tuning and improvement in a handful of trials, without needing any demonstrations. An accompanying overview video can be found in the supplementary material and at thi URL: https://sites.google.com/view/ptr-final/

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 54 of its 63 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

54 of 63 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation6 May
  2. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulation6 Aug
  3. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills15 Apr
  4. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning27 Oct

Other references · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2018Group Normalizationcited by 1,514
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2017Hindsight Experience Replaycited by 1,096
  11. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  12. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
38 more
  1. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  4. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  5. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  8. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  9. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  10. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  11. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  12. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  13. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  14. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  15. 2022Masked Visual Pre-training for Motor Controlcited by 246
  16. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  17. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  18. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  19. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  20. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  21. 2022Multi-Game Decision Transformerscited by 212
  22. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  23. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  24. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 176
  25. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
  26. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  27. 2020Visual Imitation Made Easycited by 111
  28. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
  29. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
  30. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  31. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  32. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52
  33. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  34. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  35. 2021Lifelong Robotic Reinforcement Learning by Retaining Experiencescited by 34
  36. 2021Towards Deeper Deep Reinforcement Learning with Spectral Normalizationcited by 19
  37. 2022Model-Based Offline Meta-Reinforcement Learning with Regularizationcited by 14
  38. 2020Offline Meta Learning of Explorationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 20; 2024: 31; 2025: 14; 2026: 11.

78 papers.

Sort:

By area:

  1. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  2. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  3. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  4. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  5. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  6. 2023Large Language Models for Robotics: A Surveycited by 132
  7. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  8. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 117
  9. 2023On Bringing Robots Homecited by 101
  10. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  11. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  12. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
38 more
  1. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  2. 2024RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)cited by 68
  3. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
  4. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
  5. 2023Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasetscited by 56
  6. 2025Hume: Introducing System-2 Thinking in Visual-Language-Action Modelcited by 56
  7. 2023Imitation Bootstrapped Reinforcement Learningcited by 54
  8. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  9. 2023Robot Learning with Sensorimotor Pre-trainingcited by 51
  10. 2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learningcited by 49
  11. 2024SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deploymentcited by 48
  12. 2023Finetuning Offline World Models in the Real Worldcited by 39
  13. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  14. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
  15. 2023Robotic Offline RL from Internet Videos via Value-Function Pre-Trainingcited by 31
  16. 2024Adaptive Mobile Manipulation for Articulated Objects In the Open Worldcited by 30
  17. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Modelscited by 30
  18. 2024GeRM: A Generalist Robotic Model with Mixture-of-experts for Quadruped Robotcited by 28
  19. 2025Self-Improving Embodied Foundation Modelscited by 28
  20. 2023MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulationcited by 27
  21. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  22. 2024Affordance-Guided Reinforcement Learning via Visual Promptingcited by 20
  23. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  24. 2024DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learningcited by 19
  25. 2023Scene-Driven Multimodal Knowledge Graph Construction for Embodied AIcited by 17
  26. 2024Discrete Policy: Learning Disentangled Action Space for Multi-Task Robotic Manipulationcited by 17
  27. 2022Contrastive Value Learning: Implicit Models for Simple Offline RLcited by 15
  28. 2023Merging Decision Transformers: Weight Averaging for Forming Multi-Task Policiescited by 15
  29. 2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
  30. 2024Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulationcited by 14
  31. 2023FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicingcited by 13
  32. 2024Offline Multi-task Transfer RL with Representational Penalizationcited by 13
  33. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 13
  34. 2024Diffusion Imitation from Observationcited by 12
  35. 2025Towards Embodiment Scaling Laws in Robot Locomotioncited by 12
  36. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 12
  37. 2023Robot Fleet Learning via Policy Mergingcited by 11
  38. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 9

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  2. 2022Implicit Offline Reinforcement Learning via Supervised Learning
  3. 2024D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
  4. 2023Small Dataset, Big Gains: Enhancing Reinforcement Learning by Offline Pre-Training with Model Based Augmentation
  5. 2022Contextual Transformer for Offline Meta Reinforcement Learning
  6. 2023Robotic Manipulation Datasets for Offline Compositional Reinforcement Learning
  7. 2026Cross-Embodiment Offline Reinforcement Learning for Heterogeneous Robot Datasets
  8. 2024MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning