# FusionVision: A comprehensive approach of 3D object reconstruction and segmentation from RGB-D cameras using YOLO and fast segment anything

arXiv: https://arxiv.org/abs/2403.00175 · HTML: https://mlsanity.com/paper/2403.00175/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 68 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Builds on (most cited first, 20 of 20)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [A Survey on Instance Segmentation: State of the art](https://mlsanity.com/paper/2007.00047/) (2020)
- [The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot](https://mlsanity.com/paper/2306.16623/) (2023)
- [Optimizing the Dice Score and Jaccard Index for Medical Image Segmentation: Theory & Practice](https://mlsanity.com/paper/1911.01685/) (2019)
- [DepthTrack : Unveiling the Power of RGBD Tracking](https://mlsanity.com/paper/2108.13962/) (2021)
- [Computer-Vision Benchmark Segment-Anything Model (SAM) in Medical Images: Accuracy in 12 Datasets](https://mlsanity.com/paper/2304.09324/) (2023)
- [Recent progress in semantic image segmentation](https://mlsanity.com/paper/1809.10198/) (2018)
- [Segment Anything is A Good Pseudo-label Generator for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2305.01275/) (2023)
- [RGB-D Object Detection and Semantic Segmentation for Autonomous Manipulation in Clutter](https://mlsanity.com/paper/1810.00818/) (2018)
- [Attention Swin U-Net: Cross-Contextual Attention Mechanism for Skin Lesion Segmentation](https://mlsanity.com/paper/2210.16898/) (2022)
- [NMS-Loss: Learning with Non-Maximum Suppression for Crowded Pedestrian Detection](https://mlsanity.com/paper/2106.02426/) (2021)
- [A Comprehensive Review of Modern Object Segmentation Approaches](https://mlsanity.com/paper/2301.07499/) (2023)
- [Beyond mAP: Towards better evaluation of instance segmentation](https://mlsanity.com/paper/2207.01614/) (2022)
- [Generalizing AUC Optimization to Multiclass Classification for Audio Segmentation With Limited Training Data](https://mlsanity.com/paper/2110.14425/) (2021)

## Built on by (most cited first, 1 of 1)

- [PerfCam: Digital Twinning for Production Lines Using 3D Gaussian Splatting and Vision Models](https://mlsanity.com/paper/2504.18165/) (2025)

## Abstract (arXiv)

In the realm of computer vision, the integration of advanced techniques into the processing of RGB-D camera inputs poses a significant challenge, given the inherent complexities arising from diverse environmental conditions and varying object appearances. Therefore, this paper introduces FusionVision, an exhaustive pipeline adapted for the robust 3D segmentation of objects in RGB-D imagery. Traditional computer vision systems face limitations in simultaneously capturing precise object boundaries and achieving high-precision object detection on depth map as they are mainly proposed for RGB cameras. To address this challenge, FusionVision adopts an integrated approach by merging state-of-the-art object detection techniques, with advanced instance segmentation methods. The integration of these components enables a holistic (unified analysis of information obtained from both color textitRGB and depth textitD channels) interpretation of RGB-D data, facilitating the extraction of comprehensive and accurate object information. The proposed FusionVision pipeline employs YOLO for identifying objects within the RGB image domain. Subsequently, FastSAM, an innovative semantic segmentation model, is applied to delineate object boundaries, yielding refined segmentation masks. The synergy between these components and their integration into 3D scene understanding ensures a cohesive fusion of object detection and segmentation, enhancing overall precision in 3D object segmentation. The code and pre-trained models are publicly available at https://github.com/safouaneelg/FusionVision/.