# Space-Time Memory Network for Sounding Object Localization in Videos

arXiv: https://arxiv.org/abs/2111.05526 · HTML: https://mlsanity.com/paper/2111.05526/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 10 Nov 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [Listen to Look: Action Recognition by Previewing Audio](https://mlsanity.com/paper/1912.04487/) (2019)
- [The Sound of Motions](https://mlsanity.com/paper/1904.05979/) (2019)
- [Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2007.10558/) (2020)
- [SCSampler: Sampling Salient Clips from Video for Efficient Action Recognition](https://mlsanity.com/paper/1904.04289/) (2019)
- [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020)
- [Learning to Separate Object Sounds by Watching Unlabeled Video](https://mlsanity.com/paper/1804.01665/) (2018)
- [Music Gesture for Visual Sound Separation](https://mlsanity.com/paper/2004.09476/) (2020)
- [Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching](https://mlsanity.com/paper/2010.05466/) (2020)
- [Look, Listen, and Act: Towards Audio-Visual Embodied Navigation](https://mlsanity.com/paper/1912.11684/) (2019)
- [Self-supervised Moving Vehicle Tracking with Stereo Sound](https://mlsanity.com/paper/1910.11760/) (2019)
- [Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation](https://mlsanity.com/paper/2104.02026/) (2021)
- [Watch, Listen and Tell: Multi-modal Weakly Supervised Dense Event Captioning](https://mlsanity.com/paper/1909.09944/) (2019)
- [Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data](https://mlsanity.com/paper/2006.01595/) (2020)
- [Can audio-visual integration strengthen robustness under multimodal attacks?](https://mlsanity.com/paper/2104.02000/) (2021)
- [Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning](https://mlsanity.com/paper/1804.05448/) (2018)

## Built on by (most cited first, 10 of 10)

- [Egocentric Audio-Visual Object Localization](https://mlsanity.com/paper/2303.13471/) (2023)
- [Audio-Visual Class-Incremental Learning](https://mlsanity.com/paper/2308.11073/) (2023)
- [Sound Source Localization is All about Cross-Modal Alignment](https://mlsanity.com/paper/2309.10724/) (2023)
- [Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization](https://mlsanity.com/paper/2505.11217/) (2025)
- [Can CLIP Help Sound Source Localization?](https://mlsanity.com/paper/2311.04066/) (2023)
- [Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment](https://mlsanity.com/paper/2407.13676/) (2024)
- [Unveiling Visual Biases in Audio-Visual Localization Benchmarks](https://mlsanity.com/paper/2409.06709/) (2024)
- [Hearing and Seeing Through CLIP: A Framework for Self-Supervised Sound Source Localization](https://mlsanity.com/paper/2505.05343/) (2025)
- [SOUPLE: Enhancing Audio-Visual Localization and Segmentation with Learnable Prompt Contexts](https://mlsanity.com/paper/2603.22732/) (2026)
- [SceneBind: Binding What and Where Across Vision, Audio and Language](https://mlsanity.com/paper/2607.15265/) (2026)

## Abstract (arXiv)

Leveraging temporal synchronization and association within sight and sound is an essential step towards robust localization of sounding objects. To this end, we propose a space-time memory network for sounding object localization in videos. It can simultaneously learn spatio-temporal attention over both uni-modal and cross-modal representations from audio and visual modalities. We show and analyze both quantitatively and qualitatively the effectiveness of incorporating spatio-temporal learning in localizing audio-visual objects. We demonstrate that our approach generalizes over various complex audio-visual scenes and outperforms recent state-of-the-art methods.