mlsanityData to 14 Oct 2026

Learning on the Job: Self-Rewarding Offline-to-Online Finetuning for Industrial Insertion of Novel Connectors from Vision

arXiv 2210.15206 ↗First posted 27 Oct 2022cs.RO cs.LGcited by 13

On the map: Synthetic data and self-improvement for LLMs → Self-evolving LLM agents

also in 1 more
Abstract (arXiv)

Learning-based methods in robotics hold the promise of generalization, but what can be done if a learned policy does not generalize to a new situation? In principle, if an agent can at least evaluate its own success (i.e., with a reward classifier that generalizes well even when the policy does not), it could actively practice the task and finetune the policy in this situation. We study this problem in the setting of industrial insertion tasks, such as inserting connectors in sockets and setting screws. Existing algorithms rely on precise localization of the connector or socket and carefully managed physical setups, such as assembly lines, to succeed at the task. But in unstructured environments such as homes or even some industrial settings, robots cannot rely on precise localization and may be tasked with previously unseen connectors. Offline reinforcement learning on a variety of connector insertion tasks is a potential solution, but what if the robot is tasked with inserting previously unseen connector? In such a scenario, we will still need methods that can robustly solve such tasks with online practice. One of the main observations we make in this work is that, with a suitable representation learning and domain generalization approach, it can be significantly easier for the reward function to generalize to a new but structurally similar task (e.g., inserting a new type of connector) than for the policy. This means that a learned reward function can be used to facilitate the finetuning of the robot's policy in situations where the policy fails to generalize in zero shot, but the reward function generalizes successfully. We show that such an approach can be instantiated in the real world, pretrained on 50 different connectors, and successfully finetuned to new connectors via the learned reward function. Videos can be viewed at https://sites.google.com/view/learningonthejob

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Self-evolving LLM agents, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 71 of its 93 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

71 of 93 references.

References · 71

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  9. 2015Trust Region Policy Optimizationcited by 2,580
  10. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  11. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,329
  12. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
38 more
  1. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,200
  4. 2018Sanity Checks for Saliency Mapscited by 1,156
  5. 2017Hindsight Experience Replaycited by 1,136
  6. 2016Deep Variational Information Bottleneckcited by 930
  7. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  9. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  11. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  12. 2014Deep Domain Confusion: Maximizing for Domain Invariancecited by 600
  13. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  14. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  15. 2016Domain Separation Networkscited by 320
  16. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  17. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
  18. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  19. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  20. 2016Variational Intrinsic Controlcited by 267
  21. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  22. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  23. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  24. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  25. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  26. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 174
  27. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  28. 2016Value Iteration Networkscited by 160
  29. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 153
  30. 2017Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Graspingcited by 144
  31. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
  32. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  33. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  34. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 122
  35. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  36. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 84
  37. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  38. 2020Deep Reinforcement and InfoMax Learningcited by 79

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4; 2024: 3; 2025: 5.

13 papers.

Sort:

By area:

  1. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 108
  2. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 26
  3. 2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learningcited by 12
  4. 2024MILES: Making Imitation Learning Easy with Self-Supervisioncited by 10
  5. 2024Simple Ingredients for Offline Reinforcement Learningcited by 4
  6. 2025SHaRe-RL: Structured, Interactive Reinforcement Learning for Contact-Rich Industrial Assembly Taskscited by 3
  7. 2025Failure Forecasting Boosts Robustness of Sim2Real Rhythmic Insertion Policiescited by 2
  8. 2023Robot Skill Generalization via Keypoint Integrated Soft Actor-Critic Gaussian Mixture Modelscited by 1
  9. 2023PolyFit: A Peg-in-hole Assembly Framework for Unseen Polygon Shapes via Sim-to-real Adaptationcited by 1
  10. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspectivecited by 1
  11. 2024Context-Based Meta Reinforcement Learning for Robust and Adaptable Peg-in-Hole Assembly Taskscited by 0
  12. 2025Zero-Shot Peg Insertion: Identifying Mating Holes and Estimating SE(2) Poses with Vision-Language Modelscited by 0
1 more
  1. 2025EasyInsert: A Data-Efficient and Generalizable Insertion Policycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning for Robotic Insertion of Flexible Cables in Industrial Settings
  2. 2023MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation
  3. 2026Behavioral Cloning for Robotic Connector Assembly: An Empirical Study
  4. 2023Offline Skill Generalization via Task and Motion Planning
  5. 2022How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation
  6. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning
  7. 2025Safe Learning for Contact-Rich Robot Tasks: A Survey from Classical Learning-Based Methods to Safe Foundation Models
  8. 2024Cognitive Manipulation: Semi-supervised Visual Representation and Classroom-to-real Reinforcement Learning for Assembly in Semi-structured Environments