# PolyFit: A Peg-in-hole Assembly Framework for Unseen Polygon Shapes via Sim-to-real Adaptation

arXiv: https://arxiv.org/abs/2312.02531 · HTML: https://mlsanity.com/paper/2312.02531/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 5 Dec 2023.

## Builds on (most cited first, 20 of 20)

- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience](https://mlsanity.com/paper/1810.05687/) (2018)
- [Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards](https://mlsanity.com/paper/1906.05841/) (2019)
- [Auto-Tuned Sim-to-Real Transfer](https://mlsanity.com/paper/2104.07662/) (2021)
- [Sim2Real Transfer for Reinforcement Learning without Dynamics Randomization](https://mlsanity.com/paper/2002.11635/) (2020)
- [Reinforcement Learning on Variable Impedance Controller for High-Precision Robotic Assembly](https://mlsanity.com/paper/1903.01066/) (2019)
- [Offline Meta-Reinforcement Learning for Industrial Insertion](https://mlsanity.com/paper/2110.04276/) (2021)
- [Meta-Reinforcement Learning for Robotic Industrial Insertion Tasks](https://mlsanity.com/paper/2004.14404/) (2020)
- [Variable Compliance Control for Robotic Peg-in-Hole Assembly: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2008.10224/) (2020)
- [InsertionNet -- A Scalable Solution for Insertion](https://mlsanity.com/paper/2104.14223/) (2021)
- [Extrinsic Contact Sensing with Relative-Motion Tracking from Distributed Tactile Measurements](https://mlsanity.com/paper/2103.08108/) (2021)
- [Learning on the Job: Self-Rewarding Offline-to-Online Finetuning for Industrial Insertion of Novel Connectors from Vision](https://mlsanity.com/paper/2210.15206/) (2022)
- [Fast robust peg-in-hole insertion with continuous visual servoing](https://mlsanity.com/paper/2011.06399/) (2020)
- [Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across Wide Task Spaces](https://mlsanity.com/paper/2105.11283/) (2021)
- [InsertionNet 2.0: Minimal Contact Multi-Step Insertion Using Multimodal Multiview Sensory Input](https://mlsanity.com/paper/2203.01153/) (2022)
- [Transferable Force-Torque Dynamics Model for Peg-in-hole Task](https://mlsanity.com/paper/1912.00260/) (2019)
- [Contact Skill Imitation Learning for Robot-Independent Assembly Programming](https://mlsanity.com/paper/1908.06272/) (2019)
- [Learning to Fill the Seam by Vision: Sub-millimeter Peg-in-hole on Unseen Shapes in Real World](https://mlsanity.com/paper/2204.07776/) (2022)

## Built on by (most cited first, 1 of 1)

- [A Visual Reinforcement Learning-Based Separate Primitive Policy for Peg-in-Hole Tasks](https://mlsanity.com/paper/2504.14820/) (2025)

## Abstract (arXiv)

The study addresses the foundational and challenging task of peg-in-hole assembly in robotics, where misalignments caused by sensor inaccuracies and mechanical errors often result in insertion failures or jamming. This research introduces PolyFit, representing a paradigm shift by transitioning from a reinforcement learning approach to a supervised learning methodology. PolyFit is a Force/Torque (F/T)-based supervised learning framework designed for 5-DoF peg-in-hole assembly. It utilizes F/T data for accurate extrinsic pose estimation and adjusts the peg pose to rectify misalignments. Extensive training in a simulated environment involves a dataset encompassing a diverse range of peg-hole shapes, extrinsic poses, and their corresponding contact F/T readings. To enhance extrinsic pose estimation, a multi-point contact strategy is integrated into the model input, recognizing that identical F/T readings can indicate different poses. The study proposes a sim-to-real adaptation method for real-world application, using a sim-real paired dataset to enable effective generalization to complex and unseen polygon shapes. PolyFit achieves impressive peg-in-hole success rates of 97.3% and 96.3% for seen and unseen shapes in simulations, respectively. Real-world evaluations further demonstrate substantial success rates of 86.7% and 85.0%, highlighting the robustness and adaptability of the proposed method.