# Online Object Model Reconstruction and Reuse for Lifelong Improvement of Robot Manipulation

arXiv: https://arxiv.org/abs/2109.13910 · HTML: https://mlsanity.com/paper/2109.13910/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2021.

## Built on by (most cited first, 10 of 10)

- [SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields](https://mlsanity.com/paper/2211.09786/) (2022)
- [Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangement](https://mlsanity.com/paper/2307.04751/) (2023)
- [3DSGrasp: 3D Shape-Completion for Robotic Grasp](https://mlsanity.com/paper/2301.00866/) (2023)
- [You Only Look at One: Category-Level Object Representations for Pose Estimation From a Single Example](https://mlsanity.com/paper/2305.12626/) (2023)
- [Mono-STAR: Mono-camera Scene-level Tracking and Reconstruction](https://mlsanity.com/paper/2301.13244/) (2023)
- [Task-agnostic Lifelong Robot Learning with Retrieval-based Weighted Local Adaptation](https://mlsanity.com/paper/2410.02995/) (2024)
- [Physics-Based Object 6D-Pose Estimation during Non-Prehensile Manipulation](https://mlsanity.com/paper/2211.13572/) (2022)
- [Self-Supervised Learning of Object Segmentation from Unlabeled RGB-D Videos](https://mlsanity.com/paper/2304.04325/) (2023)
- [Tabletop Transparent Scene Reconstruction via Epipolar-Guided Optical Flow with Monocular Depth Completion Prior](https://mlsanity.com/paper/2310.09956/) (2023)
- [ISAR: A Benchmark for Single- and Few-Shot Object Instance Segmentation and Re-Identification](https://mlsanity.com/paper/2311.02734/) (2023)

## Abstract (arXiv)

This work proposes a robotic pipeline for picking and constrained placement of objects without geometric shape priors. Compared to recent efforts developed for similar tasks, where every object was assumed to be novel, the proposed system recognizes previously manipulated objects and performs online model reconstruction and reuse. Over a lifelong manipulation process, the system keeps learning features of objects it has interacted with and updates their reconstructed models. Whenever an instance of a previously manipulated object reappears, the system aims to first recognize it and then register its previously reconstructed model given the current observation. This step greatly reduces object shape uncertainty allowing the system to even reason for parts of objects, which are currently not observable. This also results in better manipulation efficiency as it reduces the need for active perception of the target object during manipulation. To get a reusable reconstructed model, the proposed pipeline adopts: i) TSDF for object representation, and ii) a variant of the standard particle filter algorithm for pose estimation and tracking of the partial object model. Furthermore, an effective way to construct and maintain a dataset of manipulated objects is presented. A sequence of real-world manipulation experiments is performed. They show how future manipulation tasks become more effective and efficient by reusing reconstructed models of previously manipulated objects, which were generated during their prior manipulation, instead of treating objects as novel every time.