mlsanityData to 7 Oct 2026

Lifelong Robotic Reinforcement Learning by Retaining Experiences

arXiv 2109.09180 ↗First posted 19 Sep 2021cs.LG cs.AI cs.ROcited by 34

Not on the map. Why?

Abstract (arXiv)

Multi-task learning ideally allows robots to acquire a diverse repertoire of useful skills. However, many multi-task reinforcement learning efforts assume the robot can collect data from all tasks at all times. In reality, the tasks that the robot learns arrive sequentially, depending on the user and the robot's current environment. In this work, we study a practical sequential multi-task RL problem that is motivated by the practical constraints of physical robotic systems, and derive an approach that effectively leverages the data and policies learned for previous tasks to cumulatively grow the robot's skill-set. In a series of simulated robotic manipulation experiments, our approach requires less than half the samples than learning each task from scratch, while avoiding impractical round-robin data collection. On a Franka Emika Panda robot arm, our approach incrementally learns ten challenging tasks, including bottle capping and block insertion.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 46 of its 73 references that are in the map. First posted on arXiv 19 Sep 2021.

Builds on

46 of 73 references.

References · 46

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2016Progressive Neural Networkscited by 1,332
  5. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2018Experience Replay for Continual Learningcited by 1,074
  8. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  9. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  11. 2017Deep Reinforcement Learning that Matterscited by 727
  12. 2018Learning Dexterous In-Hand Manipulationcited by 643
34 more
  1. 2019Gradient based sample selection for online continual learningcited by 568
  2. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  3. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  4. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  5. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  6. 2015Policy Distillationcited by 414
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  8. 2016Learning to reinforcement learncited by 369
  9. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  10. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  11. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  12. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  13. 2018Selective Experience Replay for Lifelong Learningcited by 239
  14. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  15. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  16. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 176
  17. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  18. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  19. 2018Learning to Walk via Deep Reinforcement Learningcited by 153
  20. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 150
  21. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 138
  22. 2016Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfercited by 124
  23. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  24. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  25. 2018Kickstarting Deep Reinforcement Learningcited by 75
  26. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  27. 2018How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experimentscited by 56
  28. 2019Meta Reinforcement Learning for Sim-to-real Domain Adaptationcited by 49
  29. 2020Rapidly Adaptable Legged Robots via Evolutionary Meta-Learningcited by 42
  30. 2018Few-Shot Goal Inference for Visuomotor Learning and Planningcited by 30
  31. 2020The Effectiveness of Memory Replay in Large Scale Continual Learningcited by 26
  32. 2020Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learningcited by 23
  33. 2020MELD: Meta-Reinforcement Learning from Images via Latent State Modelscited by 22
  34. 2018Importance Weighted Transfer of Samples in Reinforcement Learningcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 10; 2024: 7; 2025: 4; 2026: 5.

34 papers.

Sort:

By area:

  1. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  2. 2023LOTUS: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discoverycited by 63
  3. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 56
  4. 2023Learning and Adapting Agile Locomotion Skills by Transferring Experiencecited by 25
  5. 2022You Only Live Once: Single-Life Reinforcement Learningcited by 23
  6. 2023Causal Reinforcement Learning: A Surveycited by 22
  7. 2024Online Continual Learning For Interactive Instruction Following Agentscited by 21
  8. 2025A Survey of Continual Reinforcement Learningcited by 16
  9. 2022On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learningcited by 15
  10. 2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
  11. 2026CLARE: Continual Learning for Vision-Language-Action Models via Autonomous Adapter Routing and Expansioncited by 14
  12. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 13
22 more
  1. 2024Multi-Task Reinforcement Learning for Quadrotorscited by 12
  2. 2022Continual Learning from Demonstration of Robotics Skillscited by 11
  3. 2022Relay Hindsight Experience Replay: Self-Guided Continual Reinforcement Learning for Sequential Object Manipulation Tasks with Sparse Rewardscited by 10
  4. 2023Sharing Lifelong Reinforcement Learning Knowledge via Modulating Maskscited by 10
  5. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 9
  6. 2025Dynamic Mixture of Progressive Parameter-Efficient Expert Library for Lifelong Robot Learningcited by 8
  7. 2025DRAE: Dynamic Retrieval-Augmented Expert Networks for Lifelong Learning and Task Adaptation in Roboticscited by 8
  8. 2023Adapt On-the-Go: Behavior Modulation for Single-Life Robot Deploymentcited by 7
  9. 2023Scalable and Efficient Continual Learning from Demonstration via a Hypernetwork-generated Stable Dynamics Modelcited by 6
  10. 2022Forgetting and Imbalance in Robot Lifelong Learning with Off-policy Datacited by 5
  11. 2022Exploring the Intersection between Neural Architecture Search and Continual Learningcited by 4
  12. 2024DynaMITE-RL: A Dynamic Model for Improved Temporal Meta-Reinforcement Learningcited by 3
  13. 2022Provably Efficient Lifelong Reinforcement Learning with Linear Function Approximationcited by 2
  14. 2023Continual Visual Reinforcement Learning with A Life-Long World Modelcited by 2
  15. 2023Scaling Distributed Multi-task Reinforcement Learning with Experience Sharingcited by 1
  16. 2023Modifying RL Policies with Imagined Actions: How Predictable Policies Can Enable Users to Perform Novel Taskscited by 1
  17. 2024Task-agnostic Lifelong Robot Learning with Retrieval-based Weighted Local Adaptationcited by 1
  18. 2026LiMoDE: Rethinking Lifelong Robot Manipulation from a Mixture-of-Dynamic-Experts Perspectivecited by 1
  19. 2023Evaluating Pretrained models for Deployable Lifelong Learningcited by 0
  20. 2024Imagining In-distribution States: How Predictable Robot Behavior Can Enable User Control Over Learned Policiescited by 0
  21. 2026Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learningcited by 0
  22. 2026CDCP: Conditional Diffusion Model with Contextual Prompts for Multi-task Offline Safe Reinforcement Learningcited by 0