# Self-supervised Learning for Precise Pick-and-place without Object Model

arXiv: https://arxiv.org/abs/2006.08373 · HTML: https://mlsanity.com/paper/2006.08373/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2020.

## Built on by (most cited first, 21 of 21)

- [VIMA: General Robot Manipulation with Multimodal Prompts](https://mlsanity.com/paper/2210.03094/) (2022)
- [Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation](https://mlsanity.com/paper/2112.05124/) (2021)
- [Visual Room Rearrangement](https://mlsanity.com/paper/2103.16544/) (2021)
- [SpeedFolding: Learning Efficient Bimanual Folding of Garments](https://mlsanity.com/paper/2208.10552/) (2022)
- [Equivariant Transporter Network](https://mlsanity.com/paper/2202.09400/) (2022)
- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [UGG: Unified Generative Grasping](https://mlsanity.com/paper/2311.16917/) (2023)
- [Self-Supervised Euphemism Detection and Identification for Content Moderation](https://mlsanity.com/paper/2103.16808/) (2021)
- [Leveraging Symmetries in Pick and Place](https://mlsanity.com/paper/2308.07948/) (2023)
- [Imitation Learning for Nonprehensile Manipulation through Self-Supervised Learning Considering Motion Speed](https://mlsanity.com/paper/2206.10289/) (2022)
- [Robot Skill Adaptation via Soft Actor-Critic Gaussian Mixture Models](https://mlsanity.com/paper/2111.13129/) (2021)
- [Multi-Task Learning with Sequence-Conditioned Transporter Networks](https://mlsanity.com/paper/2109.07578/) (2021)
- [World Robot Challenge 2020 -- Partner Robot: A Data-Driven Approach for Room Tidying with Mobile Manipulator](https://mlsanity.com/paper/2207.10106/) (2022)
- [NIFT: Neural Interaction Field and Template for Object Manipulation](https://mlsanity.com/paper/2210.10992/) (2022)
- [Tactile-Sensitive NewtonianVAE for High-Accuracy Industrial Connector Insertion](https://mlsanity.com/paper/2203.05955/) (2022)
- [Learning Vision-based Robotic Manipulation Tasks Sequentially in Offline Reinforcement Learning Settings](https://mlsanity.com/paper/2301.13450/) (2023)
- [Good Grasps Only: A data engine for self-supervised fine-tuning of pose estimation using grasp poses for verification](https://mlsanity.com/paper/2409.11512/) (2024)
- [Efficient, Self-Supervised Human Pose Estimation with Inductive Prior Tuning](https://mlsanity.com/paper/2311.02815/) (2023)
- [Compact Multi-Object Placement Using Adjacency-Aware Reinforcement Learning](https://mlsanity.com/paper/2404.10632/) (2024)
- [Precise Pick-and-Place using Score-Based Diffusion Networks](https://mlsanity.com/paper/2409.09725/) (2024)
- [Dynamic Knowledge Graphs as Semantic Memory Model for Industrial Robots](https://mlsanity.com/paper/2101.01099/) (2021)

## Abstract (arXiv)

Flexible pick-and-place is a fundamental yet challenging task within robotics, in particular due to the need of an object model for a simple target pose definition. In this work, the robot instead learns to pick-and-place objects using planar manipulation according to a single, demonstrated goal state. Our primary contribution lies within combining robot learning of primitives, commonly estimated by fully-convolutional neural networks, with one-shot imitation learning. Therefore, we define the place reward as a contrastive loss between real-world measurements and a task-specific noise distribution. Furthermore, we design our system to learn in a self-supervised manner, enabling real-world experiments with up to 25000 pick-and-place actions. Then, our robot is able to place trained objects with an average placement error of 2.7 (0.2) mm and 2.6 (0.8)deg. As our approach does not require an object model, the robot is able to generalize to unknown objects while keeping a precision of 5.9 (1.1) mm and 4.1 (1.2)deg. We further show a range of emerging behaviors: The robot naturally learns to select the correct object in the presence of multiple object types, precisely inserts objects within a peg game, picks screws out of dense clutter, and infers multiple pick-and-place actions from a single goal state.