# Co-Fusion: Real-time Segmentation, Tracking and Fusion of Multiple Objects

arXiv: https://arxiv.org/abs/1706.06629 · HTML: https://mlsanity.com/paper/1706.06629/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2017.

## Built on by (most cited first, 30 of 47)

- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects](https://mlsanity.com/paper/2303.14158/) (2023)
- [Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2101.06894/) (2021)
- [BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models](https://mlsanity.com/paper/2108.00516/) (2021)
- [vMAP: Vectorised Object Mapping for Neural Field SLAM](https://mlsanity.com/paper/2302.01838/) (2023)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems](https://mlsanity.com/paper/2106.14386/) (2021)
- [3D Siamese Voxel-to-BEV Tracker for Sparse Point Clouds](https://mlsanity.com/paper/2111.04426/) (2021)
- [Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networks](https://mlsanity.com/paper/2108.01176/) (2021)
- [Continuous Scene Representations for Embodied AI](https://mlsanity.com/paper/2203.17251/) (2022)
- [BodySLAM: Joint Camera Localisation, Mapping, and Human Motion Tracking](https://mlsanity.com/paper/2205.02301/) (2022)
- [DDN-SLAM: Real-time Dense Dynamic Neural Implicit SLAM](https://mlsanity.com/paper/2401.01545/) (2024)
- [DenseSplat: Densifying Gaussian Splatting SLAM with Neural Radiance Prior](https://mlsanity.com/paper/2502.09111/) (2025)
- [4DTAM: Non-Rigid Tracking and Mapping via Dynamic Surface Gaussians](https://mlsanity.com/paper/2505.22859/) (2025)
- [DROID-SLAM in the Wild](https://mlsanity.com/paper/2603.19076/) (2026)
- [Learning to Complete Object Shapes for Object-level Mapping in Dynamic Scenes](https://mlsanity.com/paper/2208.05067/) (2022)
- [3D LiDAR Mapping in Dynamic Environments Using a 4D Implicit Neural Representation](https://mlsanity.com/paper/2405.03388/) (2024)
- [Semantic Visual Simultaneous Localization and Mapping: A Survey](https://mlsanity.com/paper/2209.06428/) (2022)
- [A Simple Approach for Visual Rearrangement: 3D Mapping and Semantic Search](https://mlsanity.com/paper/2206.13396/) (2022)
- [TSDF++: A Multi-Object Formulation for Dynamic Object Tracking and Reconstruction](https://mlsanity.com/paper/2105.07468/) (2021)
- [Comparison of modern open-source visual SLAM approaches](https://mlsanity.com/paper/2108.01654/) (2021)
- [TwistSLAM: Constrained SLAM in Dynamic Environment](https://mlsanity.com/paper/2202.12384/) (2022)
- [Scene-level Tracking and Reconstruction without Object Priors](https://mlsanity.com/paper/2210.03815/) (2022)
- [Mono-STAR: Mono-camera Scene-level Tracking and Reconstruction](https://mlsanity.com/paper/2301.13244/) (2023)
- [Factored Neural Representation for Scene Understanding](https://mlsanity.com/paper/2304.10950/) (2023)
- [Physics-Based Rigid Body Object Tracking and Friction Filtering From RGB-D Videos](https://mlsanity.com/paper/2309.15703/) (2023)
- [Visual-Inertial Multi-Instance Dynamic SLAM with Object-level Relocalisation](https://mlsanity.com/paper/2208.04274/) (2022)
- [TivNe-SLAM: Dynamic Mapping and Tracking via Time-Varying Neural Radiance Fields](https://mlsanity.com/paper/2310.18917/) (2023)
- [NeuSurfEmb: A Complete Pipeline for Dense Correspondence-based 6D Object Pose Estimation without CAD Models](https://mlsanity.com/paper/2407.12207/) (2024)
- [BYE: Build Your Encoder with One Sequence of Exploration Data for Long-Term Dynamic Scene Understanding](https://mlsanity.com/paper/2412.02449/) (2024)

## Abstract (arXiv)

In this paper we introduce Co-Fusion, a dense SLAM system that takes a live stream of RGB-D images as input and segments the scene into different objects (using either motion or semantic cues) while simultaneously tracking and reconstructing their 3D shape in real time. We use a multiple model fitting approach where each object can move independently from the background and still be effectively tracked and its shape fused over time using only the information from pixels associated with that object label. Previous attempts to deal with dynamic scenes have typically considered moving regions as outliers, and consequently do not model their shape or track their motion over time. In contrast, we enable the robot to maintain 3D models for each of the segmented objects and to improve them over time through fusion. As a result, our system can enable a robot to maintain a scene description at the object level which has the potential to allow interactions with its working environment; even in the case of dynamic scenes.