Learning Sim-to-Real Dense Object Descriptors for Robotic Manipulation
Not on the map. Why?
Abstract (arXiv)
It is crucial to address the following issues for ubiquitous robotics manipulation applications: (a) vision-based manipulation tasks require the robot to visually learn and understand the object with rich information like dense object descriptors; and (b) sim-to-real transfer in robotics aims to close the gap between simulated and real data. In this paper, we present Sim-to-Real Dense Object Nets (SRDONs), a dense object descriptor that not only understands the object via appropriate representation but also maps simulated and real data to a unified feature space with pixel consistency. We proposed an object-to-object matching method for image pairs from different scenes and different domains. This method helps reduce the effort of training data from real-world by taking advantage of public datasets, such as GraspNet. With sim-to-real object representation consistency, our SRDONs can serve as a building block for a variety of sim-to-real manipulation tasks. We demonstrate in experiments that pre-trained SRDONs significantly improve performances on unseen objects and unseen visual environments for various robotic tasks with zero real-world training.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 18 of its 22 references that are in the map. First posted on arXiv 18 Apr 2023.
Builds on
18 of 22 references.
References · 18
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2015Domain-Adversarial Training of Neural Networkscited by 3,044
- 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,329
- 2019Solving Rubik's Cube with a Robot Handcited by 734
- 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 413
- 2016Domain Separation Networkscited by 320
- 2016Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networkscited by 308
- 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
- 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 142
- 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 91
- 2020Unseen Object Instance Segmentation for Robotic Environmentscited by 85
- 2020RetinaGAN: An Object-aware Approach to Sim-to-Real Transfercited by 70
6 more
- 2020Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentationcited by 64
- 2020Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Datacited by 51
- 2020Perspectives on Sim2Real Transfer for Robotics: A Summary of the R:SS 2020 Workshopcited by 25
- 2021What Can I Do Here? Learning New Skills by Imagining Visual Affordancescited by 23
- 2019Self-Supervised Sim-to-Real Adaptation for Visual Robotic Manipulationcited by 19
- 2021Supervised Training of Dense Object Nets using Optimal Descriptors for Industrial Robotic Applicationscited by 4
Built on by
Citing papers per year: 2024: 2.
2 papers.