mlsanityData to 7 Oct 2026

COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning

arXiv 2010.14500 ↗First posted 27 Oct 2020cs.LG cs.ROcited by 83

On the map: Offline RL and imitation learning from data → Real-world robot offline RL

Abstract (arXiv)

Reinforcement learning has been applied to a wide variety of robotics problems, but most of such applications involve collecting data from scratch for each new task. Since the amount of robot data we can collect for any single task is limited by time and cost considerations, the learned behavior is typically narrow: the policy can only execute the task in a handful of scenarios that it was trained on. What if there was a way to incorporate a large amount of prior data, either from previously solved tasks or from unsupervised or undirected environment interaction, to extend and generalize learned behaviors? While most prior work on extending robotic skills using pre-collected data focuses on building explicit hierarchies or skill decompositions, we show in this paper that we can reuse prior data to extend new skills simply through dynamic programming. We show that even when the prior data does not actually succeed at solving the new task, it can still be utilized for learning a better policy, by providing the agent with a broader understanding of the mechanics of its environment. We demonstrate the effectiveness of our approach by chaining together several behaviors seen in prior datasets for solving a new task, with our hardest experimental setting involving composing four robotic skills in a row: picking, placing, drawer opening, and grasping, where a +1/0 sparse reward is provided only on task completion. We train our policies in an end-to-end fashion, mapping high-dimensional image observations to low-level robot control commands, and present results in both simulated and real world domains. Additional materials and source code can be found on our project website: https://sites.google.com/view/cog-rl

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 51 references that are in the map. First posted on arXiv 27 Oct 2020.

Builds on

34 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019Scaling data-driven robotics with reward sketching and batch reinforcement learning26 Sep

Other references · 33

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  9. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  11. 2018Learning Dexterous In-Hand Manipulationcited by 643
  12. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
21 more
  1. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  2. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  3. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  4. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  5. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  6. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  7. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 124
  8. 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 121
  9. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  10. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 114
  11. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  12. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  13. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  14. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  15. 2019Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresightcited by 34
  16. 2016Learning Dexterous Manipulation Policies from Experience and Imitationcited by 32
  17. 2016Visual Closed-Loop Control for Pouring Liquidscited by 26
  18. 2016Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Searchcited by 25
  19. 2020Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learningcited by 23
  20. 2016Path Integral Guided Policy Searchcited by 14
  21. 2018Interpretable Latent Spaces for Learning from Demonstrationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 21; 2023: 22; 2024: 17; 2025: 3; 2026: 2.

83 papers.

Sort:

By area:

  1. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  2. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  3. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  5. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  6. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  7. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  8. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  9. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  10. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  11. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  12. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learningcited by 77
38 more
  1. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 73
  2. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 68
  3. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  4. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  5. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
  6. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 48
  7. 2021Offline Reinforcement Learning with Reverse Model-based Imaginationcited by 47
  8. 2023Chat with the Environment: Interactive Multimodal Perception Using Large Language Modelscited by 42
  9. 2021Offline Meta-Reinforcement Learning for Industrial Insertioncited by 37
  10. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learningcited by 34
  11. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31
  12. 2023Robotic Offline RL from Internet Videos via Value-Function Pre-Trainingcited by 31
  13. 2022Offline Reinforcement Learning for Visual Navigationcited by 28
  14. 2021Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Studycited by 24
  15. 2022Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Taskscited by 24
  16. 2023SPRINT: Scalable Policy Pre-Training via Language Instruction Relabelingcited by 24
  17. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
  18. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  19. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  20. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learningcited by 18
  21. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  22. 2021Collect & Infer -- a fresh look at data-efficient Reinforcement Learningcited by 17
  23. 2022On the Role of Discount Factor in Offline Reinforcement Learningcited by 16
  24. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 16
  25. 2021C-Planning: An Automatic Curriculum for Learning Goal-Reaching Taskscited by 15
  26. 2023Hierarchical Imitation Learning with Vector Quantized Modelscited by 15
  27. 2024Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learningcited by 14
  28. 2022Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraintscited by 13
  29. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Datacited by 13
  30. 2022Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discoverycited by 12
  31. 2022Learning on the Job: Self-Rewarding Offline-to-Online Finetuning for Industrial Insertion of Novel Connectors from Visioncited by 11
  32. 2023Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learningcited by 11
  33. 2024D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learningcited by 11
  34. 2023The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learningcited by 10
  35. 2023Replay across Experiments: A Natural Extension of Off-Policy RLcited by 10
  36. 2024Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuningcited by 10
  37. 2023PLEX: Making the Most of the Available Data for Robotic Manipulation Pretrainingcited by 9
  38. 2024Distributional Successor Features Enable Zero-Shot Policy Optimizationcited by 8

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
  2. 2023Offline Skill Generalization via Task and Motion Planning
  3. 2022How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation
  4. 2022Implicit Offline Reinforcement Learning via Supervised Learning
  5. 2026SkillWeaver: Agentic Exploration over Neural Interaction Skills for Scalable Robot Data Generation
  6. 2021Continuous Doubly Constrained Batch Reinforcement Learning
  7. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  8. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning