mlsanityData to 7 Oct 2026

Offline Meta-Reinforcement Learning for Industrial Insertion

arXiv 2110.04276 ↗First posted 8 Oct 2021cs.ROcited by 37

On the map: Offline RL and imitation learning from data → Real-world robot offline RL

Abstract (arXiv)

Reinforcement learning (RL) can in principle let robots automatically adapt to new tasks, but current RL methods require a large number of trials to accomplish this. In this paper, we tackle rapid adaptation to new tasks through the framework of meta-learning, which utilizes past tasks to learn to adapt with a specific focus on industrial insertion tasks. Fast adaptation is crucial because prohibitively large number of on-robot trials will potentially damage hardware pieces. Additionally, effective adaptation is also feasible in that experience among different insertion applications can be largely leveraged by each other. In this setting, we address two specific challenges when applying meta-learning. First, conventional meta-RL algorithms require lengthy online meta-training. We show that this can be replaced with appropriately chosen offline data, resulting in an offline meta-RL method that only requires demonstrations and trials from each of the prior tasks, without the need to run costly meta-RL procedures online. Second, meta-RL methods can fail to generalize to new tasks that are too different from those seen at meta-training time, which poses a particular challenge in industrial applications, where high success rates are critical. We address this by combining contextual meta-learning with direct online finetuning: if the new task is similar to those seen in the prior data, then the contextual meta-learner adapts immediately, and if it is too different, it gradually adapts through finetuning. We show that our approach is able to quickly adapt to a variety of different insertion tasks, with a success rate of 100% using only a fraction of the samples needed for learning the tasks from scratch. Experiment videos and details are available at https://sites.google.com/view/offline-metarl-insertion.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 52 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

37 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning27 Oct

Other references · 36

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  8. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  9. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  11. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  12. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
24 more
  1. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  2. 2018Residual Reinforcement Learning for Robot Controlcited by 242
  3. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  4. 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 128
  5. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
  6. 2018Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learningcited by 91
  7. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  8. 2018Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLcited by 80
  9. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  10. 2018Composable Deep Reinforcement Learning for Robotic Manipulationcited by 70
  11. 2019Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewardscited by 58
  12. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52
  13. 2015Learning Contact-Rich Manipulation Skills with Guided Policy Searchcited by 37
  14. 2017Deep Reinforcement Learning for High Precision Assembly Taskscited by 37
  15. 2018A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learningcited by 35
  16. 2019Reinforcement Learning on Variable Impedance Controller for High-Precision Robotic Assemblycited by 33
  17. 2020Meta-Reinforcement Learning for Robotic Industrial Insertion Taskscited by 28
  18. 2021Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Studycited by 24
  19. 2020MELD: Meta-Reinforcement Learning from Images via Latent State Modelscited by 22
  20. 2018Learning Robotic Assembly from CADcited by 21
  21. 2016Learning from the Hindsight Plan -- Episodic MPC Improvementcited by 15
  22. 2021InsertionNet -- A Scalable Solution for Insertioncited by 15
  23. 2020Offline Meta Learning of Explorationcited by 12
  24. 2020FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularizationcited by 6

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 6; 2023: 11; 2024: 12; 2025: 7.

37 papers.

Sort:

By area:

  1. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  2. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  3. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  4. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  5. 2023Multi-Stage Cable Routing through Hierarchical Imitation Learningcited by 59
  6. 2024RLDG: Robotic Generalist Policy Distillation via Reinforcement Learningcited by 42
  7. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
  8. 2024TacDiffusion: Force-domain Diffusion Policy for Precise Tactile Manipulationcited by 23
  9. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 16
  10. 2024Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learningcited by 16
  11. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  12. 2025Touch begins where vision ends: Generalizable policies for contact-rich manipulationcited by 13
25 more
  1. 2022Learning on the Job: Self-Rewarding Offline-to-Online Finetuning for Industrial Insertion of Novel Connectors from Visioncited by 11
  2. 2025Rethinking Bimanual Robotic Manipulation: Learning with Decoupled Interaction Frameworkcited by 11
  3. 2024GenCHiP: Generating Robot Policy Code for High-Precision and Contact-Rich Manipulation Taskscited by 9
  4. 2024MILES: Making Imitation Learning Easy with Self-Supervisioncited by 9
  5. 2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptionscited by 8
  6. 2024Data-driven Machinery Fault Diagnosis: A Comprehensive Reviewcited by 7
  7. 2023Seq2Seq Imitation Learning for Tactile Feedback-based Manipulationcited by 6
  8. 2023Robotic Manipulation Datasets for Offline Compositional Reinforcement Learningcited by 5
  9. 2023Multi-level Reasoning for Robotic Assembly: From Sequence Inference to Contact Selectioncited by 5
  10. 2022Reinforcement learning with Demonstrations from Mismatched Task under Sparse Rewardcited by 4
  11. 2023Environment Transformer and Policy Optimization for Model-Based Offline Reinforcement Learningcited by 4
  12. 2025Self-evolved Imitation Learning in Simulated Worldcited by 4
  13. 2023Unleash Model Potential: Bootstrapped Meta Self-supervised Learningcited by 2
  14. 2025Robotic Paper Wrapping by Learning Force Controlcited by 2
  15. 2022Enhanced Meta Reinforcement Learning using Demonstrations in Sparse Reward Environmentscited by 1
  16. 2022Accelerating Laboratory Automation Through Robot Skill Learning For Sample Scrapingcited by 1
  17. 2023Learning Excavation of Rigid Objects with Offline Reinforcement Learningcited by 1
  18. 2023Diff-Transfer: Model-based Robotic Manipulation Skill Transfer via Differentiable Physics Simulationcited by 1
  19. 2024Automating Deformable Gasket Assemblycited by 1
  20. 2022Learning A Simulation-based Visual Policy for Real-world Peg In Unseen Holescited by 0
  21. 2023Bootstrapping Adaptive Human-Machine Interfaces with Offline Reinforcement Learningcited by 0
  22. 2023PolyFit: A Peg-in-hole Assembly Framework for Unseen Polygon Shapes via Sim-to-real Adaptationcited by 0
  23. 2024Automating Transfer of Robot Task Plans using Functorial Data Migrationscited by 0
  24. 2025Zero-Shot Peg Insertion: Identifying Mating Holes and Estimating SE(2) Poses with Vision-Language Modelscited by 0
  25. 2025EasyInsert: A Data-Efficient and Generalizable Insertion Policycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Robotic Control Using Model Based Meta Adaption
  2. 2025Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data
  3. 2023Offline Meta Reinforcement Learning with In-Distribution Online Adaptation
  4. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  5. 2022Model-Based Offline Meta-Reinforcement Learning with Regularization
  6. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardware
  7. 2021Efficient Robotic Manipulation Through Offline-to-Online Reinforcement Learning and Goal-Aware State Information
  8. 2024MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning