mlsanityData to 14 Oct 2026

Mix and Localize: Localizing Sound Sources in Mixtures

arXiv 2211.15058 ↗First posted 28 Nov 2022cs.CVcited by 54

Not on the map. Why?

Abstract (arXiv)

We present a method for simultaneously localizing multiple sound sources within a visual scene. This task requires a model to both group a sound mixture into individual sources, and to associate them with a visual signal. Our method jointly solves both tasks at once, using a formulation inspired by the contrastive random walk of Jabri et al. We create a graph in which images and separated sounds correspond to nodes, and train a random walker to transition between nodes from different modalities with high return probability. The transition probabilities for this walk are determined by an audio-visual similarity metric that is learned by our model. We show through experiments with musical instruments and human speech that our model can successfully localize multiple sounds, outperforming other self-supervised methods. Project site: https://hxixixh.github.io/mix-and-localize

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Builds on 43 of its 54 references that are in the map. First posted on arXiv 28 Nov 2022.

Builds on

43 of 54 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  5. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  6. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  7. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  8. 2017Look, Listen and Learncited by 458
  9. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  10. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  11. 2018The Sound of Pixelscited by 259
  12. 2017Objects that Soundcited by 256
31 more
  1. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249
  2. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
  3. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  4. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 212
  5. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 191
  6. 2018Learning to Localize Sound Source in Visual Scenescited by 189
  7. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 188
  8. 2021Localizing Visual Sounds the Hard Waycited by 170
  9. 2018Tracking Emerges by Colorizing Videoscited by 166
  10. 2019The Sound of Motionscited by 146
  11. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 146
  12. 2015Visually Indicated Soundscited by 128
  13. 2018Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separationcited by 125
  14. 2020Multiple Sound Sources Localization from Coarse to Finecited by 118
  15. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  16. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 109
  17. 2020Evaluating Weakly Supervised Object Localization Methods Rightcited by 105
  18. 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
  19. 2020Labelling unlabelled videos from scratch with multi-modal self-supervisioncited by 100
  20. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  21. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  22. 2018Seeing Voices and Hearing Faces: Cross-modal biometric matchingcited by 60
  23. 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
  24. 2020VisualEchoes: Spatial Image Representation Learning through Echolocationcited by 55
  25. 2019Self-Supervised Audio-Visual Co-Segmentationcited by 44
  26. 2021There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledgecited by 44
  27. 2020Telling Left from Right: Learning Spatial Correspondence of Sight and Soundcited by 43
  28. 2022Learning Pixel Trajectories with Multiscale Contrastive Random Walkscited by 36
  29. 2021Move2Hear: Active Audio-Visual Source Separationcited by 35
  30. 2021Visual Scene Graphs for Audio Source Separationcited by 29
  31. 2021Structure from Silence: Learning Scene Structure from Ambient Soundcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 18; 2024: 17; 2025: 16; 2026: 2.

54 papers.

Sort:

By area:

  1. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 95
  2. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 85
  3. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  4. 2023Audio-Visual Grouping Network for Sound Localization from Mixturescited by 52
  5. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  6. 2023Egocentric Audio-Visual Object Localizationcited by 40
  7. 2023Improving Audio-Visual Segmentation with Bidirectional Generationcited by 37
  8. 2023Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentationcited by 34
  9. 2023Audio-Visual Class-Incremental Learningcited by 33
  10. 2023Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modelingcited by 24
  11. 2023Class-Incremental Grouping Network for Continual Audio-Visual Learningcited by 22
  12. 2023A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognitioncited by 21
38 more
  1. 2023Language-Guided Audio-Visual Source Separation via Trimodal Consistencycited by 20
  2. 2023Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipationcited by 20
  3. 2024T-VSL: Text-Guided Visual Sound Source Localization in Mixturescited by 19
  4. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  5. 2023Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parsercited by 18
  6. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  7. 2024Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledgecited by 14
  8. 2024MA-AVT: Modality Alignment for Parameter-Efficient Audio-Visual Transformerscited by 11
  9. 2024CPM: Class-conditional Prompting Machine for Audio-visual Segmentationcited by 11
  10. 2023CAD -- Contextual Multi-modal Alignment for Dynamic AVQAcited by 10
  11. 2024Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for Audio-Visual Source Localizationcited by 10
  12. 2023Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarioscited by 8
  13. 2024Unsupervised Audio-Visual Segmentation with Modality Alignmentcited by 7
  14. 2023Learning to Taste: A Multimodal Wine Datasetcited by 6
  15. 2025Do Audio-Visual Segmentation Models Truly Segment Sounding Objects?cited by 6
  16. 2025Revisiting Audio-Visual Segmentation with Vision-Centric Transformercited by 6
  17. 2025Object-aware Sound Source Localization via Audio-Visual Scene Understandingcited by 5
  18. 2025Implicit Counterfactual Learning for Audio-Visual Segmentationcited by 5
  19. 2025Complementary and Contrastive Learning for Audio-Visual Segmentationcited by 5
  20. 2024CLIP-Powered TASS: Target-Aware Single-Stream Network for Audio-Visual Question Answeringcited by 4
  21. 2024A Critical Assessment of Visual Sound Source Localization Models Including Negative Audiocited by 4
  22. 2024Aligning Audio-Visual Joint Representations with an Agentic Workflowcited by 4
  23. 2025CCStereo: Audio-Visual Contextual and Contrastive Learning for Binaural Audio Generationcited by 4
  24. 2025What's Making That Sound Right Now? Video-centric Audio-Visual Localizationcited by 4
  25. 2024Cross Pseudo-Labeling for Semi-Supervised Audio-Visual Source Localizationcited by 3
  26. 2024Enhancing Sound Source Localization via False Negative Eliminationcited by 3
  27. 2024Multi-scale Multi-instance Visual Sound Localization and Segmentationcited by 3
  28. 2024Unveiling Visual Biases in Audio-Visual Localization Benchmarkscited by 3
  29. 2025Self-Supervised Spatial Correspondence Across Modalitiescited by 3
  30. 2025MultiModal Action Conditioned Video Generationcited by 3
  31. 2024A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learningcited by 2
  32. 2025UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsingcited by 2
  33. 2023Induction Network: Audio-Visual Modality Gap-Bridging for Self-Supervised Sound Source Localizationcited by 1
  34. 2023STELLA: Continual Audio-Video Pre-training with Spatio-Temporal Localized Alignmentcited by 1
  35. 2025OpenAVS: Training-Free Open-Vocabulary Audio Visual Segmentation with Foundational Modelscited by 1
  36. 2025DEL: Dense Event Localization for Multi-modal Audio-Visual Understandingcited by 1
  37. 2025Semi-Supervised Audio-Visual Video Action Recognition with Audio Source Localization Guided Mixupcited by 0
  38. 2025Enhancing Explainability with Multimodal Context Representations for Smarter Robotscited by 0

50 most cited of 54 All 54 (JSON)