# Artifacts Mapping: Multi-Modal Semantic Mapping for Object Detection and 3D Localization

arXiv: https://arxiv.org/abs/2307.01121 · HTML: https://mlsanity.com/paper/2307.01121/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 3 Jul 2023.

## Builds on (most cited first, 25 of 25)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [Multi-Task Multi-Sensor Fusion for 3D Object Detection](https://mlsanity.com/paper/2012.12397/) (2020)
- [Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping](https://mlsanity.com/paper/1910.02490/) (2019)
- [Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization](https://mlsanity.com/paper/2201.13360/) (2022)
- [3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans](https://mlsanity.com/paper/2002.06289/) (2020)
- [SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks](https://mlsanity.com/paper/1609.05130/) (2016)
- [SuMa++: Efficient LiDAR-based Semantic SLAM](https://mlsanity.com/paper/2105.11320/) (2021)
- [Fusion++: Volumetric Object-Level SLAM](https://mlsanity.com/paper/1808.08378/) (2018)
- [MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects](https://mlsanity.com/paper/1804.09194/) (2018)
- [YOLACT++: Better Real-time Instance Segmentation](https://mlsanity.com/paper/1912.06218/) (2019)
- [Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery](https://mlsanity.com/paper/1903.00268/) (2019)
- [Meaningful Maps With Object-Oriented Semantic Mapping](https://mlsanity.com/paper/1609.07849/) (2016)
- [Camera-Lidar Integration: Probabilistic sensor fusion for semantic mapping](https://mlsanity.com/paper/2007.05490/) (2020)
- [DA-RNN: Semantic Mapping with Data Associated Recurrent Neural Networks](https://mlsanity.com/paper/1703.03098/) (2017)
- [Real-time Progressive 3D Semantic Segmentation for Indoor Scene](https://mlsanity.com/paper/1804.00257/) (2018)
- [YolactEdge: Real-time Instance Segmentation on the Edge](https://mlsanity.com/paper/2012.12259/) (2020)
- [Real-Time Multi-Modal Semantic Fusion on Unmanned Aerial Vehicles with Label Propagation for Cross-Domain Adaptation](https://mlsanity.com/paper/2210.09739/) (2022)
- [Monocular SLAM Supported Object Recognition](https://mlsanity.com/paper/1506.01732/) (2015)
- [Semantic Mapping with Simultaneous Object Detection and Localization](https://mlsanity.com/paper/1810.11525/) (2018)
- [Vision-based Large-scale 3D Semantic Mapping for Autonomous Driving Applications](https://mlsanity.com/paper/2203.01087/) (2022)
- [Object-level 3D Semantic Mapping using a Network of Smart Edge Sensors](https://mlsanity.com/paper/2211.11354/) (2022)
- [WoLF: the Whole-body Locomotion Framework for Quadruped Robots](https://mlsanity.com/paper/2205.06526/) (2022)

## Abstract (arXiv)

Geometric navigation is nowadays a well-established field of robotics and the research focus is shifting towards higher-level scene understanding, such as Semantic Mapping. When a robot needs to interact with its environment, it must be able to comprehend the contextual information of its surroundings. This work focuses on classifying and localising objects within a map, which is under construction (SLAM) or already built. To further explore this direction, we propose a framework that can autonomously detect and localize predefined objects in a known environment using a multi-modal sensor fusion approach (combining RGB and depth data from an RGB-D camera and a lidar). The framework consists of three key elements: understanding the environment through RGB data, estimating depth through multi-modal sensor fusion, and managing artifacts (i.e., filtering and stabilizing measurements). The experiments show that the proposed framework can accurately detect 98% of the objects in the real sample environment, without post-processing, while 85% and 80% of the objects were mapped using the single RGBD camera or RGB + lidar setup respectively. The comparison with single-sensor (camera or lidar) experiments is performed to show that sensor fusion allows the robot to accurately detect near and far obstacles, which would have been noisy or imprecise in a purely visual or laser-based approach.