# MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects

arXiv: https://arxiv.org/abs/1804.09194 · HTML: https://mlsanity.com/paper/1804.09194/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2018.

## Built on by (most cited first, 30 of 95)

- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2210.05714/) (2022)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [Open-vocabulary Queryable Scene Representations for Real World Planning](https://mlsanity.com/paper/2209.09874/) (2022)
- [BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects](https://mlsanity.com/paper/2303.14158/) (2023)
- [Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2101.06894/) (2021)
- [BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models](https://mlsanity.com/paper/2108.00516/) (2021)
- [SimpleRecon: 3D Reconstruction Without 3D Convolutions](https://mlsanity.com/paper/2208.14743/) (2022)
- [OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Data](https://mlsanity.com/paper/2311.02873/) (2023)
- [Weakly Supervised Learning of Rigid 3D Scene Flow](https://mlsanity.com/paper/2102.08945/) (2021)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems](https://mlsanity.com/paper/2106.14386/) (2021)
- [Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networks](https://mlsanity.com/paper/2108.01176/) (2021)
- [Any6D: Model-free 6D Pose Estimation of Novel Objects](https://mlsanity.com/paper/2503.18673/) (2025)
- [Incremental 3D Semantic Scene Graph Prediction from RGB Sequences](https://mlsanity.com/paper/2305.02743/) (2023)
- [TTA-COPE: Test-Time Adaptation for Category-Level Object Pose Estimation](https://mlsanity.com/paper/2303.16730/) (2023)
- [DSP-SLAM: Object Oriented SLAM with Deep Shape Priors](https://mlsanity.com/paper/2108.09481/) (2021)
- [Audio Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2303.07522/) (2023)
- [PVO: Panoptic Visual Odometry](https://mlsanity.com/paper/2207.01610/) (2022)
- [UDA-COPE: Unsupervised Domain Adaptation for Category-level Object Pose Estimation](https://mlsanity.com/paper/2111.12580/) (2021)
- [BodySLAM: Joint Camera Localisation, Mapping, and Human Motion Tracking](https://mlsanity.com/paper/2205.02301/) (2022)
- [RayFronts: Open-Set Semantic Ray Frontiers for Online Scene Understanding and Exploration](https://mlsanity.com/paper/2504.06994/) (2025)
- [SO-SLAM: Semantic Object SLAM with Scale Proportional and Symmetrical Texture Constraints](https://mlsanity.com/paper/2109.04884/) (2021)
- [DynaMoN: Motion-Aware Fast and Robust Camera Localization for Dynamic Neural Radiance Fields](https://mlsanity.com/paper/2309.08927/) (2023)
- [An Object SLAM Framework for Association, Mapping, and High-Level Tasks](https://mlsanity.com/paper/2305.07299/) (2023)
- [Objects Can Move: 3D Change Detection by Geometric Transformation Constistency](https://mlsanity.com/paper/2208.09870/) (2022)
- [Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Models](https://mlsanity.com/paper/2409.15451/) (2024)
- [OnlineAnySeg: Online Zero-Shot 3D Segmentation by Visual Foundation Model Guided 2D Mask Merging](https://mlsanity.com/paper/2503.01309/) (2025)
- [6DOPE-GS: Online 6D Object Pose Estimation using Gaussian Splatting](https://mlsanity.com/paper/2412.01543/) (2024)
- [NeuSE: Neural SE(3)-Equivariant Embedding for Long-Term Object-Based Simultaneous Localization and Mapping](https://mlsanity.com/paper/2303.07308/) (2023)

## Abstract (arXiv)

We present MaskFusion, a real-time, object-aware, semantic and dynamic RGB-D SLAM system that goes beyond traditional systems which output a purely geometric map of a static scene. MaskFusion recognizes, segments and assigns semantic class labels to different objects in the scene, while tracking and reconstructing them even when they move independently from the camera. As an RGB-D camera scans a cluttered scene, image-based instance-level semantic segmentation creates semantic object masks that enable real-time object recognition and the creation of an object-level representation for the world map. Unlike previous recognition-based SLAM systems, MaskFusion does not require known models of the objects it can recognize, and can deal with multiple independent motions. MaskFusion takes full advantage of using instance-level semantic segmentation to enable semantic labels to be fused into an object-aware map, unlike recent semantics enabled SLAM systems that perform voxel-level semantic segmentation. We show augmented-reality applications that demonstrate the unique features of the map output by MaskFusion: instance-aware, semantic and dynamic.