# Learning to Complete Object Shapes for Object-level Mapping in Dynamic Scenes

arXiv: https://arxiv.org/abs/2208.05067 · HTML: https://mlsanity.com/paper/2208.05067/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 9 Aug 2022.

## Builds on (most cited first, 27 of 27)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks](https://mlsanity.com/paper/1609.05130/) (2016)
- [DynaSLAM: Tracking, Mapping and Inpainting in Dynamic Scenes](https://mlsanity.com/paper/1806.05620/) (2018)
- [Fusion++: Volumetric Object-Level SLAM](https://mlsanity.com/paper/1808.08378/) (2018)
- [MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects](https://mlsanity.com/paper/1804.09194/) (2018)
- [BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models](https://mlsanity.com/paper/2108.00516/) (2021)
- [MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM](https://mlsanity.com/paper/1812.07976/) (2018)
- [NodeSLAM: Neural Object Descriptors for Multi-View Shape Reconstruction](https://mlsanity.com/paper/2004.04485/) (2020)
- [CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations](https://mlsanity.com/paper/2008.02792/) (2020)
- [Co-Fusion: Real-time Segmentation, Tracking and Fusion of Multiple Objects](https://mlsanity.com/paper/1706.06629/) (2017)
- [DynaSLAM II: Tightly-Coupled Multi-Object Tracking and SLAM](https://mlsanity.com/paper/2010.07820/) (2020)
- [DSP-SLAM: Object Oriented SLAM with Deep Shape Priors](https://mlsanity.com/paper/2108.09481/) (2021)
- [EM-Fusion: Dynamic Object-Level SLAM with Probabilistic Data Association](https://mlsanity.com/paper/1904.11781/) (2019)
- [MOLTR: Multiple Object Localisation, Tracking, and Reconstruction from Monocular RGB Videos](https://mlsanity.com/paper/2012.05360/) (2020)
- [Seeing Behind Objects for 3D Multi-Object Tracking in RGB-D Sequences](https://mlsanity.com/paper/2012.08197/) (2020)
- [Deep Probabilistic Feature-metric Tracking](https://mlsanity.com/paper/2008.13504/) (2020)
- [Deep-SLAM++: Object-level RGBD SLAM based on class-specific deep shape priors](https://mlsanity.com/paper/1907.09691/) (2019)
- [Where Does It End? -- Reasoning About Hidden Surfaces by Object Intersection Constraints](https://mlsanity.com/paper/2004.04630/) (2020)

## Built on by (most cited first, 8 of 8)

- [An Object SLAM Framework for Association, Mapping, and High-Level Tasks](https://mlsanity.com/paper/2305.07299/) (2023)
- [SC-Diff: 3D Shape Completion with Latent Diffusion Models](https://mlsanity.com/paper/2403.12470/) (2024)
- [TextSLAM: Visual SLAM with Semantic Planar Text Features](https://mlsanity.com/paper/2305.10029/) (2023)
- [FuncGrasp: Learning Object-Centric Neural Grasp Functions from Single Annotated Example Object](https://mlsanity.com/paper/2402.05644/) (2024)
- [Toward General Object-level Mapping from Sparse Views with 3D Diffusion Priors](https://mlsanity.com/paper/2410.05514/) (2024)
- [Anthropomorphic Grasping with Neural Object Shape Completion](https://mlsanity.com/paper/2311.02510/) (2023)
- [HIPPo: Harnessing Image-to-3D Priors for Model-free Zero-shot 6D Pose Estimation](https://mlsanity.com/paper/2502.10606/) (2025)
- [Temporally Consistent Object 6D Pose Estimation for Robot Control](https://mlsanity.com/paper/2605.02708/) (2026)

## Abstract (arXiv)

In this paper, we propose a novel object-level mapping system that can simultaneously segment, track, and reconstruct objects in dynamic scenes. It can further predict and complete their full geometries by conditioning on reconstructions from depth inputs and a category-level shape prior with the aim that completed object geometry leads to better object reconstruction and tracking accuracy. For each incoming RGB-D frame, we perform instance segmentation to detect objects and build data associations between the detection and the existing object maps. A new object map will be created for each unmatched detection. For each matched object, we jointly optimise its pose and latent geometry representations using geometric residual and differential rendering residual towards its shape prior and completed geometry. Our approach shows better tracking and reconstruction performance compared to methods using traditional volumetric mapping or learned shape prior approaches. We evaluate its effectiveness by quantitatively and qualitatively testing it in both synthetic and real-world sequences.