# Vision-driven Compliant Manipulation for Reliable, High-Precision Assembly Tasks

arXiv: https://arxiv.org/abs/2106.14070 · HTML: https://mlsanity.com/paper/2106.14070/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 65 references that are in the map.
- First posted on arXiv 26 Jun 2021.

## Builds on (most cited first, 28 of 28)

- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control](https://mlsanity.com/paper/1812.00568/) (2018)
- [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018)
- [A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms](https://mlsanity.com/paper/1907.03146/) (2019)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains](https://mlsanity.com/paper/2007.13866/) (2020)
- [Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2009.05085/) (2020)
- [Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly](https://mlsanity.com/paper/1910.13675/) (2019)
- [Cable Manipulation with a Tactile-Reactive Gripper](https://mlsanity.com/paper/1910.02860/) (2019)
- [Analysis and Observations from the First Amazon Picking Challenge](https://mlsanity.com/paper/1601.05484/) (2016)
- [Learning Contact-Rich Manipulation Skills with Guided Policy Search](https://mlsanity.com/paper/1501.05611/) (2015)
- [kPAM 2.0: Feedback Control for Category-Level Robotic Manipulation](https://mlsanity.com/paper/2102.06279/) (2021)
- [Meta-Reinforcement Learning for Robotic Industrial Insertion Tasks](https://mlsanity.com/paper/2004.14404/) (2020)
- [Real-time Perception meets Reactive Motion Generation](https://mlsanity.com/paper/1703.03512/) (2017)
- [Learning a visuomotor controller for real world robotic grasping using simulated depth images](https://mlsanity.com/paper/1706.04652/) (2017)
- [Variable Compliance Control for Robotic Peg-in-Hole Assembly: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2008.10224/) (2020)
- [Robust, Occlusion-aware Pose Estimation for Objects Grasped by Adaptive Hands](https://mlsanity.com/paper/2003.03518/) (2020)
- [Depth-Based Object Tracking Using a Robust Gaussian Filter](https://mlsanity.com/paper/1602.06157/) (2016)
- [Probabilistic Object Tracking using a Range Camera](https://mlsanity.com/paper/1505.00241/) (2015)
- [Learning Robotic Assembly from CAD](https://mlsanity.com/paper/1803.07635/) (2018)
- [A Region-based Gauss-Newton Approach to Real-Time Monocular Multiple Object Tracking](https://mlsanity.com/paper/1807.02087/) (2018)
- [Towards Robotic Feeding: Role of Haptics in Fork-based Food Manipulation](https://mlsanity.com/paper/1804.08768/) (2018)
- [Compare Contact Model-based Control and Contact Model-free Learning: A Survey of Robotic Peg-in-hole Assembly Strategies](https://mlsanity.com/paper/1904.05240/) (2019)
- [Task-driven Perception and Manipulation for Constrained Placement of Unknown Objects](https://mlsanity.com/paper/2006.15503/) (2020)

## Built on by (most cited first, 18 of 18)

- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models](https://mlsanity.com/paper/2108.00516/) (2021)
- [You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration](https://mlsanity.com/paper/2201.12716/) (2022)
- [CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation](https://mlsanity.com/paper/2109.09163/) (2021)
- [DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulation](https://mlsanity.com/paper/2312.00583/) (2023)
- [Human-in-the-Loop Task and Motion Planning for Imitation Learning](https://mlsanity.com/paper/2310.16014/) (2023)
- [RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control](https://mlsanity.com/paper/2210.11668/) (2022)
- [6D Robotic Assembly Based on RGB-only Object Pose Estimation](https://mlsanity.com/paper/2208.12986/) (2022)
- [GenCHiP: Generating Robot Policy Code for High-Precision and Contact-Rich Manipulation Tasks](https://mlsanity.com/paper/2404.06645/) (2024)
- [GeoReF: Geometric Alignment Across Shape Variation for Category-level Object Pose Refinement](https://mlsanity.com/paper/2404.11139/) (2024)
- [Learning Sensorimotor Primitives of Sequential Manipulation Tasks from Visual Demonstrations](https://mlsanity.com/paper/2203.03797/) (2022)
- [Is Single-View Mesh Reconstruction Ready for Robotics?](https://mlsanity.com/paper/2505.17966/) (2025)
- [MResT: Multi-Resolution Sensing for Real-Time Control with Vision-Language Models](https://mlsanity.com/paper/2401.14502/) (2024)
- [DeLTa: Demonstration and Language-Guided Novel Transparent Object Manipulation](https://mlsanity.com/paper/2510.05662/) (2025)
- [Learning A Simulation-based Visual Policy for Real-world Peg In Unseen Holes](https://mlsanity.com/paper/2205.04297/) (2022)
- [Self-supervised 6-DoF Robot Grasping by Demonstration via Augmented Reality Teleoperation System](https://mlsanity.com/paper/2404.03067/) (2024)
- [Learning Generalizable 3D Manipulation With 10 Demonstrations](https://mlsanity.com/paper/2411.10203/) (2024)
- [SI-Diff: A Framework for Learning Search and High-Precision Insertion with a Force-Domain Diffusion Policy](https://mlsanity.com/paper/2605.12247/) (2026)

## Abstract (arXiv)

Highly constrained manipulation tasks continue to be challenging for autonomous robots as they require high levels of precision, typically less than 1mm, which is often incompatible with what can be achieved by traditional perception systems. This paper demonstrates that the combination of state-of-the-art object tracking with passively adaptive mechanical hardware can be leveraged to complete precision manipulation tasks with tight, industrially-relevant tolerances (0.25mm). The proposed control method closes the loop through vision by tracking the relative 6D pose of objects in the relevant workspace. It adjusts the control reference of both the compliant manipulator and the hand to complete object insertion tasks via within-hand manipulation. Contrary to previous efforts for insertion, our method does not require expensive force sensors, precision manipulators, or time-consuming, online learning, which is data hungry. Instead, this effort leverages mechanical compliance and utilizes an object agnostic manipulation model of the hand learned offline, off-the-shelf motion planning, and an RGBD-based object tracker trained solely with synthetic data. These features allow the proposed system to easily generalize and transfer to new tasks and environments. This paper describes in detail the system components and showcases its efficacy with extensive experiments involving tight tolerance peg-in-hole insertion tasks of various geometries as well as open-world constrained placement tasks.