# DSP-SLAM: Object Oriented SLAM with Deep Shape Priors

arXiv: https://arxiv.org/abs/2108.09481 · HTML: https://mlsanity.com/paper/2108.09481/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 49 references that are in the map.
- First posted on arXiv 21 Aug 2021.

## Builds on (most cited first, 30 of 33)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction](https://mlsanity.com/paper/1905.10711/) (2019)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation](https://mlsanity.com/paper/2002.10111/) (2020)
- [Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency](https://mlsanity.com/paper/1704.06254/) (2017)
- [SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks](https://mlsanity.com/paper/1609.05130/) (2016)
- [DynaSLAM: Tracking, Mapping and Inpainting in Dynamic Scenes](https://mlsanity.com/paper/1806.05620/) (2018)
- [A micro Lie theory for state estimation in robotics](https://mlsanity.com/paper/1812.01537/) (2018)
- [CubeSLAM: Monocular 3D Object SLAM](https://mlsanity.com/paper/1806.00557/) (2018)
- [SuMa++: Efficient LiDAR-based Semantic SLAM](https://mlsanity.com/paper/2105.11320/) (2021)
- [Fusion++: Volumetric Object-Level SLAM](https://mlsanity.com/paper/1808.08378/) (2018)
- [MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects](https://mlsanity.com/paper/1804.09194/) (2018)
- [QuadricSLAM: Dual Quadrics from Object Detections as Landmarks in Object-oriented SLAM](https://mlsanity.com/paper/1804.04011/) (2018)
- [Autolabeling 3D Objects with Differentiable Rendering of SDF Shape Priors](https://mlsanity.com/paper/1911.11288/) (2019)
- [MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM](https://mlsanity.com/paper/1812.07976/) (2018)
- [A Large Dataset of Object Scans](https://mlsanity.com/paper/1602.02481/) (2016)
- [NodeSLAM: Neural Object Descriptors for Multi-View Shape Reconstruction](https://mlsanity.com/paper/2004.04485/) (2020)
- [FroDO: From Detections to 3D Objects](https://mlsanity.com/paper/2005.05125/) (2020)
- [Stereo DSO: Large-Scale Direct Sparse Visual Odometry with Stereo Cameras](https://mlsanity.com/paper/1708.07878/) (2017)
- [Photometric Mesh Optimization for Video-Aligned 3D Object Reconstruction](https://mlsanity.com/paper/1903.08642/) (2019)
- [DOPS: Learning to Detect 3D Objects and Predict their 3D Shapes](https://mlsanity.com/paper/2004.01170/) (2020)
- [Real-Time Monocular Object-Model Aware Sparse SLAM](https://mlsanity.com/paper/1809.09149/) (2018)
- [Deep-SLAM++: Object-level RGBD SLAM based on class-specific deep shape priors](https://mlsanity.com/paper/1907.09691/) (2019)
- [Real-time Monocular Object SLAM](https://mlsanity.com/paper/1504.02398/) (2015)

## Built on by (most cited first, 30 of 30)

- [vMAP: Vectorised Object Mapping for Neural Field SLAM](https://mlsanity.com/paper/2302.01838/) (2023)
- [OA-SLAM: Leveraging Objects for Camera Relocalization in Visual SLAM](https://mlsanity.com/paper/2209.08338/) (2022)
- [An Object SLAM Framework for Association, Mapping, and High-Level Tasks](https://mlsanity.com/paper/2305.07299/) (2023)
- [Panoptic Mapping with Fruit Completion and Pose Estimation for Horticultural Robots](https://mlsanity.com/paper/2303.08923/) (2023)
- [RoCo:Robust Collaborative Perception By Iterative Object Matching and Pose Adjustment](https://mlsanity.com/paper/2408.00257/) (2024)
- [Open-Vocabulary Online Semantic Mapping for SLAM](https://mlsanity.com/paper/2411.15043/) (2024)
- [NeuSE: Neural SE(3)-Equivariant Embedding for Long-Term Object-Based Simultaneous Localization and Mapping](https://mlsanity.com/paper/2303.07308/) (2023)
- [Multi-view 3D Object Reconstruction and Uncertainty Modelling with Neural Shape Prior](https://mlsanity.com/paper/2306.11739/) (2023)
- [Learning to Complete Object Shapes for Object-level Mapping in Dynamic Scenes](https://mlsanity.com/paper/2208.05067/) (2022)
- [Uncertainty-aware 3D Object-Level Mapping with Deep Shape Priors](https://mlsanity.com/paper/2309.09118/) (2023)
- [Semantic Visual Simultaneous Localization and Mapping: A Survey](https://mlsanity.com/paper/2209.06428/) (2022)
- [FuncGrasp: Learning Object-Centric Neural Grasp Functions from Single Annotated Example Object](https://mlsanity.com/paper/2402.05644/) (2024)
- [Toward General Object-level Mapping from Sparse Views with 3D Diffusion Priors](https://mlsanity.com/paper/2410.05514/) (2024)
- [GSLAMOT: A Tracklet and Query Graph-based Simultaneous Locating, Mapping, and Multiple Object Tracking System](https://mlsanity.com/paper/2408.09191/) (2024)
- [Semantic Enhancement for Object SLAM with Heterogeneous Multimodal Large Language Model Agents](https://mlsanity.com/paper/2411.06752/) (2024)
- [Active 6D Pose Estimation for Textureless Objects using Multi-View RGB Frames](https://mlsanity.com/paper/2503.03726/) (2025)
- [TransPoser: Transformer as an Optimizer for Joint Object Shape and Pose Estimation](https://mlsanity.com/paper/2303.13477/) (2023)
- [ShapeICP: Iterative Category-level Object Pose and Shape Estimation from Depth](https://mlsanity.com/paper/2408.13147/) (2024)
- [TwistSLAM++: Fusing multiple modalities for accurate dynamic semantic SLAM](https://mlsanity.com/paper/2209.07888/) (2022)
- [Object-based SLAM utilizing unambiguous pose parameters considering general symmetry types](https://mlsanity.com/paper/2303.07872/) (2023)
- [Attention-Guided Lidar Segmentation and Odometry Using Image-to-Point Cloud Saliency Transfer](https://mlsanity.com/paper/2308.14332/) (2023)
- [UniQuadric: A SLAM Backend for Unknown Rigid Object 3D Tracking and Light-Weight Modeling](https://mlsanity.com/paper/2309.17036/) (2023)
- [Are All Point Clouds Suitable for Completion? Weakly Supervised Quality Evaluation Network for Point Cloud Completion](https://mlsanity.com/paper/2303.01804/) (2023)
- [Convex Hull-based Algebraic Constraint for Visual Quadric SLAM](https://mlsanity.com/paper/2503.01254/) (2025)
- [Category-level Meta-learned NeRF Priors for Efficient Object Mapping](https://mlsanity.com/paper/2503.01582/) (2025)
- [NF-SLAM: Effective, Normalizing Flow-supported Neural Field representations for object-level visual SLAM in automotive applications](https://mlsanity.com/paper/2503.11199/) (2025)
- [VOCAL: Visual Odometry via ContrAstive Learning](https://mlsanity.com/paper/2507.00243/) (2025)
- [DSP-SLAM++: A Unified Framework for Multi-Class, High-Fidelity Object SLAM in the Wild](https://mlsanity.com/paper/2606.25953/) (2026)
- [Hydra++: Real-Time Hierarchical 3D Scene Graph Construction With Object-Level Shape Estimation](https://mlsanity.com/paper/2607.09455/) (2026)
- [CODA: Depth-Aligned Scene Completion and Object Decomposition from a Single RGB-D Image](https://mlsanity.com/paper/2609.25654/) (2026)

## Abstract (arXiv)

We propose DSP-SLAM, an object-oriented SLAM system that builds a rich and accurate joint map of dense 3D models for foreground objects, and sparse landmark points to represent the background. DSP-SLAM takes as input the 3D point cloud reconstructed by a feature-based SLAM system and equips it with the ability to enhance its sparse map with dense reconstructions of detected objects. Objects are detected via semantic instance segmentation, and their shape and pose is estimated using category-specific deep shape embeddings as priors, via a novel second order optimization. Our object-aware bundle adjustment builds a pose-graph to jointly optimize camera poses, object locations and feature points. DSP-SLAM can operate at 10 frames per second on 3 different input modalities: monocular, stereo, or stereo+LiDAR. We demonstrate DSP-SLAM operating at almost frame rate on monocular-RGB sequences from the Friburg and Redwood-OS datasets, and on stereo+LiDAR sequences on the KITTI odometry dataset showing that it achieves high-quality full object reconstructions, even from partial observations, while maintaining a consistent global map. Our evaluation shows improvements in object pose and shape reconstruction with respect to recent deep prior-based reconstruction methods and reductions in camera tracking drift on the KITTI dataset.