mlsanityData to 7 Oct 2026

Active Audio-Visual Separation of Dynamic Sound Sources

arXiv 2202.00850 ↗First posted 2 Feb 2022cs.CV cs.LG cs.SDcited by 16

Not on the map. Why?

Abstract (arXiv)

We explore active audio-visual separation for dynamic sound sources, where an embodied agent moves intelligently in a 3D environment to continuously isolate the time-varying audio stream being emitted by an object of interest. The agent hears a mixed stream of multiple audio sources (e.g., multiple people conversing and a band playing music at a noisy party). Given a limited time budget, it needs to extract the target sound accurately at every step using egocentric audio-visual observations. We propose a reinforcement learning agent equipped with a novel transformer memory that learns motion policies to control its camera and microphone to recover the dynamic target audio, using self-attention to make high-quality estimates for current timesteps and also simultaneously improve its past estimates. Using highly realistic acoustic SoundSpaces simulations in real-world scanned Matterport3D environments, we show that our model is able to learn efficient behavior to carry out continuous separation of a dynamic audio target. Project: https://vision.cs.utexas.edu/projects/active-av-dynamic-separation/.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 43 of its 83 references that are in the map. First posted on arXiv 2 Feb 2022.

Builds on

43 of 83 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2016Layer Normalizationcited by 5,579
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  9. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  10. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  11. 2018SDR - half-baked or well done?cited by 420
  12. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
31 more
  1. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  2. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  3. 2018The Sound of Pixelscited by 258
  4. 2020Attention is All You Need in Speech Separationcited by 184
  5. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 115
  6. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  7. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  8. 2020Music Gesture for Visual Sound Separationcited by 112
  9. 2019Co-Separating Sounds of Visual Objectscited by 110
  10. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 107
  11. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 106
  12. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 95
  13. 2020Semantic Audio-Visual Navigationcited by 79
  14. 2020Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separationcited by 67
  15. 2021Memory-Augmented Reinforcement Learning for Image-Goal Navigationcited by 64
  16. 20182.5D Visual Soundcited by 63
  17. 2014Deeply learned face representations are sparse, selective, and robustcited by 56
  18. 2020Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Soundscited by 55
  19. 2019Recursive Visual Sound Separation Using Minus-Plus Netcited by 53
  20. 2021Move2Hear: Active Audio-Visual Source Separationcited by 35
  21. 2020Listen to What You Want: Neural Network-based Universal Sound Selectorcited by 27
  22. 2019My lips are concealed: Audio-visual speech enhancement through obstructionscited by 26
  23. 2020FaceFilter: Audio-visual speech separation using still imagescited by 25
  24. 2020Exploiting Scene-specific Features for Object Goal Navigationcited by 20
  25. 2021SpecTNT: a Time-Frequency Transformer for Music Audiocited by 15
  26. 2021Environment Predictive Coding for Embodied Agentscited by 12
  27. 2021TransMask: A Compact and Fast Speech Separation Model Based on Transformercited by 9
  28. 2019WildMix Dataset and Spectro-Temporal Transformer Model for Monoaural Audio Source Separationcited by 7
  29. 2020Learning to Visually Navigate in Photorealistic Environments Without any Supervisioncited by 6
  30. 2013Vision-Guided Robot Hearingcited by 4
  31. 2020Temporal-Spatial Neural Filter: Direction Informed End-to-End Multi-channel Target Speech Separationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 9; 2024: 3; 2025: 1; 2026: 1.

16 papers.

Sort:

By area:

  1. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  2. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  3. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
  4. 2023Measuring Acoustics with Collaborative Multiple Agentscited by 21
  5. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  6. 2023AdVerb: Visually Guided Audio Dereverberationcited by 15
  7. 2023Chat2Map: Efficient Scene Mapping from Multi-Ego Conversationscited by 8
  8. 2023RealImpact: A Dataset of Impact Sound Fields for Real Objectscited by 8
  9. 2023Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videoscited by 8
  10. 2023LAVSS: Location-Guided Audio-Visual Spatial Audio Separationcited by 8
  11. 2023Novel-View Acoustic Synthesis from 3D Reconstructed Roomscited by 6
  12. 2023Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separationcited by 4
4 more
  1. 2024Cross Pseudo-Labeling for Semi-Supervised Audio-Visual Source Localizationcited by 3
  2. 2024ActiveRIR: Active Audio-Visual Exploration for Acoustic Environment Modelingcited by 2
  3. 2025Rethinking Audio-Visual Adversarial Vulnerability from Temporal and Modality Perspectivescited by 2
  4. 2026Materialistic RIR: Material Conditioned Realistic RIR Generationcited by 0