# The Sound of Bounding-Boxes

arXiv: https://arxiv.org/abs/2203.15991 · HTML: https://mlsanity.com/paper/2203.15991/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Builds on (most cited first, 18 of 18)

- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Dilated Residual Networks](https://mlsanity.com/paper/1705.09914/) (2017)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021)
- [The Sound of Motions](https://mlsanity.com/paper/1904.05979/) (2019)
- [Learning to Separate Object Sounds by Watching Unlabeled Video](https://mlsanity.com/paper/1804.01665/) (2018)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Music Gesture for Visual Sound Separation](https://mlsanity.com/paper/2004.09476/) (2020)
- [Co-Separating Sounds of Visual Objects](https://mlsanity.com/paper/1904.07750/) (2019)
- [The Conversation: Deep Audio-Visual Speech Enhancement](https://mlsanity.com/paper/1804.04121/) (2018)
- [Self-Supervised Audio-Visual Co-Segmentation](https://mlsanity.com/paper/1904.09013/) (2019)
- [Visual Scene Graphs for Audio Source Separation](https://mlsanity.com/paper/2109.11955/) (2021)
- [Do We Need Sound for Sound Source Localization?](https://mlsanity.com/paper/2007.05722/) (2020)

## Abstract (arXiv)

In the task of audio-visual sound source separation, which leverages visual information for sound source separation, identifying objects in an image is a crucial step prior to separating the sound source. However, existing methods that assign sound on detected bounding boxes suffer from a problem that their approach heavily relies on pre-trained object detectors. Specifically, when using these existing methods, it is required to predetermine all the possible categories of objects that can produce sound and use an object detector applicable to all such categories. To tackle this problem, we propose a fully unsupervised method that learns to detect objects in an image and separate sound source simultaneously. As our method does not rely on any pre-trained detector, our method is applicable to arbitrary categories without any additional annotation. Furthermore, although being fully unsupervised, we found that our method performs comparably in separation accuracy.