# AcousticFusion: Fusing Sound Source Localization to Visual SLAM in Dynamic Environments

arXiv: https://arxiv.org/abs/2108.01246 · HTML: https://mlsanity.com/paper/2108.01246/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 28 references that are in the map.
- First posted on arXiv 3 Aug 2021.

## Builds on (most cited first, 12 of 12)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [DynaSLAM: Tracking, Mapping and Inpainting in Dynamic Scenes](https://mlsanity.com/paper/1806.05620/) (2018)
- [DS-SLAM: A Semantic Visual SLAM towards Dynamic Environments](https://mlsanity.com/paper/1809.08379/) (2018)
- [OpenVSLAM: A Versatile Visual SLAM Framework](https://mlsanity.com/paper/1910.01122/) (2019)
- [FlowFusion: Dynamic Dense RGB-D SLAM Based on Optical Flow](https://mlsanity.com/paper/2003.05102/) (2020)
- [Variational Bayesian Inference for Audio-Visual Tracking of Multiple Speakers](https://mlsanity.com/paper/1809.10961/) (2018)
- [Multimotion Visual Odometry (MVO): Simultaneous Estimation of Camera and Third-Party Motions](https://mlsanity.com/paper/1808.00274/) (2018)
- [Reverberant Sound Localization with a Robot Head Based on Direct-Path Relative Transfer Function](https://mlsanity.com/paper/2012.03574/) (2020)
- [Online Localization and Tracking of Multiple Moving Speakers in Reverberant Environments](https://mlsanity.com/paper/1809.10936/) (2018)
- [3D Localization of a Sound Source Using Mobile Microphone Arrays Referenced by SLAM](https://mlsanity.com/paper/2007.11079/) (2020)
- [RigidFusion: Robot Localisation and Mapping in Environments with Large Dynamic Rigid Objects](https://mlsanity.com/paper/2010.10841/) (2020)

## Built on by (most cited first, 2 of 2)

- [A Comprehensive Survey on Video Saliency Detection with Auditory Information: the Audio-visual Consistency Perceptual is the Key!](https://mlsanity.com/paper/2206.13390/) (2022)
- [Active propulsion noise shaping for multi-rotor aircraft localization](https://mlsanity.com/paper/2402.17289/) (2024)

## Abstract (arXiv)

Dynamic objects in the environment, such as people and other agents, lead to challenges for existing simultaneous localization and mapping (SLAM) approaches. To deal with dynamic environments, computer vision researchers usually apply some learning-based object detectors to remove these dynamic objects. However, these object detectors are computationally too expensive for mobile robot on-board processing. In practical applications, these objects output noisy sounds that can be effectively detected by on-board sound source localization. The directional information of the sound source object can be efficiently obtained by direction of sound arrival (DoA) estimation, but depth estimation is difficult. Therefore, in this paper, we propose a novel audio-visual fusion approach that fuses sound source direction into the RGB-D image and thus removes the effect of dynamic obstacles on the multi-robot SLAM system. Experimental results of multi-robot SLAM in different dynamic environments show that the proposed method uses very small computational resources to obtain very stable self-localization results.