A Simple Approach to Continual Learning by Transferring Skill Parameters
Not on the map. Why?
Abstract (arXiv)
In order to be effective general purpose machines in real world environments, robots not only will need to adapt their existing manipulation skills to new circumstances, they will need to acquire entirely new skills on-the-fly. A great promise of continual learning is to endow robots with this ability, by using their accumulated knowledge and experience from prior skills. We take a fresh look at this problem, by considering a setting in which the robot is limited to storing that knowledge and experience only in the form of learned skill policies. We show that storing skill policies, careful pre-training, and appropriately choosing when to transfer those skill policies is sufficient to build a continual learner in the context of robotic manipulation. We analyze which conditions are needed to transfer skills in the challenging Meta-World simulation benchmark. Using this analysis, we introduce a pair-wise metric relating skills that allows us to predict the effectiveness of skill transfer between tasks, and use it to reduce the problem of continual learning to curriculum selection. Given an appropriate curriculum, we show how to continually acquire robotic manipulation skills without forgetting, and using far fewer samples than needed to train them from scratch.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 86 of its 136 references that are in the map. First posted on arXiv 19 Oct 2021.
Builds on
86 of 136 references.
References · 86
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
- 2016Progressive Neural Networkscited by 1,332
- 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
- 2015End-to-End Training of Deep Visuomotor Policiescited by 847
- 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
- 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
- 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
- 2019Solving Rubik's Cube with a Robot Handcited by 699
38 more
- 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
- 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
- 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
- 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
- 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
- 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
- 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 277
- 2016Deep Visual Foresight for Planning Robot Motioncited by 276
- 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
- 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
- 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
- 2018Visual Reinforcement Learning with Imagined Goalscited by 248
- 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
- 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
- 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
- 2014Robots that can adapt like animalscited by 203
- 2018An Introduction to Deep Reinforcement Learningcited by 201
- 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
- 2016Learning to Navigate in Complex Environmentscited by 192
- 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
- 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
- 2018Learning to Walk via Deep Reinforcement Learningcited by 153
- 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 152
- 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
- 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
- 2018Residual Policy Learningcited by 146
- 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 138
- 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
- 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 121
- 2018Zero-Shot Visual Imitationcited by 114
- 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
- 2018Motion Planning Among Dynamic, Decision-Making Agents with Deep Reinforcement Learningcited by 101
- 2016Sim-to-Real Robot Learning from Pixels with Progressive Netscited by 89
- 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
- 2017Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learningcited by 81
- 2018Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLcited by 80
- 2020One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RLcited by 79
- 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
50 most cited of 86 All 86 (JSON)
Built on by
Citing papers per year: 2023: 2.
2 papers.