mlsanityData to 7 Oct 2026

Localizing Visual Sounds the Hard Way

arXiv 2104.02691 ↗First posted 6 Apr 2021cs.CV eess.AS eess.IVcited by 169

Not on the map. Why?

Abstract (arXiv)

The objective of this work is to localize sound sources that are visible in a video without using manual annotations. Our key technical contribution is to show that, by training the network to explicitly discriminate challenging image fragments, even for images that do contain the object emitting the sound, we can significantly boost the localization performance. We do so elegantly by introducing a mechanism to mine hard samples and add them to a contrastive learning formulation automatically. We show that our algorithm achieves state-of-the-art performance on the popular Flickr SoundNet dataset. Furthermore, we introduce the VGG-Sound Source (VGG-SS) benchmark, a new set of annotations for the recently-introduced VGG-Sound dataset, where the sound sources visible in each video clip are explicitly marked with bounding box annotations. This dataset is 20 times larger than analogous existing ones, contains 5K videos spanning over 200 categories, and, differently from Flickr SoundNet, is video-based. On VGG-SS, we also show that our algorithm achieves state-of-the-art performance against several baselines.

Key facts. Cited by 169 papers in the map (data through 7 Oct 2026). Builds on 23 of its 39 references that are in the map. First posted on arXiv 6 Apr 2021.

Builds on

23 of 39 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  5. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  6. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 577
  7. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  8. 2018Scale-recurrent Network for Deep Image Deblurringcited by 332
  9. 2018The Sound of Pixelscited by 258
  10. 2017Objects that Soundcited by 254
  11. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  12. 2018Learning to Localize Sound Source in Visual Scenescited by 188
11 more
  1. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  2. 2019Video Representation Learning by Dense Predictive Codingcited by 175
  3. 2019The VIA Annotation Software for Images, Audio and Videocited by 166
  4. 2020Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsingcited by 146
  5. 2019The Sound of Motionscited by 145
  6. 2020Multiple Sound Sources Localization from Coarse to Finecited by 118
  7. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  8. 20182.5D Visual Soundcited by 63
  9. 2019Self-supervised Moving Vehicle Tracking with Stereo Soundcited by 63
  10. 2019Self-Supervised Audio-Visual Co-Segmentationcited by 44
  11. 2018On Attention Modules for Audio-Visual Synchronizationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 30; 2023: 45; 2024: 41; 2025: 30; 2026: 13.

169 papers.

Sort:

By area:

  1. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  2. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  3. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  4. 2023Multi-modal Learning with Missing Modality via Shared-Specific Feature Modellingcited by 159
  5. 2022Audio-Visual Segmentationcited by 148
  6. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  7. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  8. 2022Localizing Visual Sounds the Easy Waycited by 81
  9. 2023Audio-Visual Segmentation with Semanticscited by 73
  10. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 72
  11. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  12. 2023AVSegFormer: Audio-Visual Segmentation with Transformercited by 66
38 more
  1. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  2. 2023Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentationcited by 54
  3. 2023Audio-Visual Grouping Network for Sound Localization from Mixturescited by 52
  4. 2023CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentationcited by 52
  5. 2024POP-3D: Open-Vocabulary 3D Occupancy Prediction from Imagescited by 52
  6. 2023Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learningcited by 46
  7. 2023Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baselinecited by 45
  8. 2023Annotation-free Audio-Visual Segmentationcited by 44
  9. 2022Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisationcited by 43
  10. 2023Sound to Visual Scene Generation by Audio-to-Visual Latent Alignmentcited by 41
  11. 2021Improving Multi-Modal Learning with Uni-Modal Teacherscited by 40
  12. 2023Egocentric Audio-Visual Object Localizationcited by 40
  13. 2021Audio-Visual Synchronisation in the wildcited by 39
  14. 2023STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Eventscited by 39
  15. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 38
  16. 2023Connecting Multi-modal Contrastive Representationscited by 36
  17. 2023Improving Audio-Visual Segmentation with Bidirectional Generationcited by 36
  18. 2023Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizercited by 35
  19. 2023Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentationcited by 33
  20. 2023Audio-Visual Class-Incremental Learningcited by 33
  21. 2021Class-aware Sounding Objects Localization via Audiovisual Correspondencecited by 32
  22. 2022Learning Sound Localization Better From Semantically Similar Samplescited by 32
  23. 2023Audio-aware Query-enhanced Transformer for Audio-Visual Segmentationcited by 32
  24. 2023Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semanticscited by 32
  25. 2023Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learningcited by 32
  26. 2024Ref-AVS: Refer and Segment Objects in Audio-Visual Scenescited by 32
  27. 2022Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenescited by 31
  28. 2023BAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledgecited by 30
  29. 2023Sound Source Localization is All about Cross-Modal Alignmentcited by 30
  30. 2021Self-supervised object detection from audio-visual correspondencecited by 29
  31. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  32. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 28
  33. 2022Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localizationcited by 27
  34. 2022Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Languagecited by 27
  35. 2024Audio-Visual Segmentation via Unlabeled Frame Exploitationcited by 27
  36. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
  37. 2024Towards Open-Vocabulary Audio-Visual Event Localizationcited by 26
  38. 2023Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentationcited by 25

50 most cited of 169 All 169 (JSON)