mlsanityData to 14 Oct 2026

The Sound of Pixels

arXiv 1804.03160 ↗First posted 9 Apr 2018cs.CV cs.SD eess.AScited by 259

Not on the map. Why?

Abstract (arXiv)

We introduce PixelPlayer, a system that, by leveraging large amounts of unlabeled videos, learns to locate image regions which produce sounds and separate the input sounds into a set of components that represents the sound from each pixel. Our approach capitalizes on the natural synchronization of the visual and audio modalities to learn models that jointly parse sounds and images, without requiring additional manual supervision. Experimental results on a newly collected MUSIC dataset show that our proposed Mix-and-Separate framework outperforms several baselines on source separation. Qualitative results suggest our model learns to ground sounds in vision, enabling applications such as independently adjusting the volume of sound sources.

Key facts. Cited by 259 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 64; 2022: 48; 2023: 48; 2024: 48; 2025: 31; 2026: 20.

259 papers.

Sort:

By area:

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  2. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  3. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 264
  4. 2023A Cookbook of Self-Supervised Learningcited by 250
  5. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 191
  6. 2021Localizing Visual Sounds the Hard Waycited by 170
  7. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 153
  8. 2022Audio-Visual Segmentationcited by 151
  9. 2021Taming Visually Guided Sound Generationcited by 136
  10. 2021NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenescited by 119
  11. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  12. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 107
38 more
  1. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  2. 2022Localizing Visual Sounds the Easy Waycited by 82
  3. 2021Robust Audio-Visual Instance Discriminationcited by 80
  4. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 75
  5. 2023Audio-Visual Segmentation with Semanticscited by 75
  6. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 73
  7. 2021Cross Modal Retrieval with Querybank Normalisationcited by 72
  8. 2021Positive Sample Propagation along the Audio-Visual Event Linecited by 69
  9. 2022Audio Self-supervised Learning: A Surveycited by 65
  10. 2022Learning Neural Acoustic Fieldscited by 65
  11. 2021Learning 3D Semantic Segmentation with only 2D Image Supervisioncited by 60
  12. 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
  13. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  14. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  15. 2021Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptionscited by 53
  16. 2023Audio-Visual Grouping Network for Sound Localization from Mixturescited by 52
  17. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  18. 2022Contrastive Positive Sample Propagation along the Audio-Visual Event Linecited by 50
  19. 2023AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesiscited by 49
  20. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
  21. 2023Annotation-free Audio-Visual Segmentationcited by 46
  22. 2022Visual Acoustic Matchingcited by 45
  23. 2021There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledgecited by 44
  24. 2022Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisationcited by 44
  25. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  26. 2023Separate Anything You Describecited by 43
  27. 2021Improving Multi-Modal Learning with Uni-Modal Teacherscited by 41
  28. 2023The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objectscited by 41
  29. 2023STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Eventscited by 41
  30. 2021Audio-Visual Synchronisation in the wildcited by 40
  31. 2023Egocentric Audio-Visual Object Localizationcited by 40
  32. 2021MM-Pyramid: Multimodal Pyramid Attentional Network for Audio-Visual Event Localization and Video Parsingcited by 39
  33. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 39
  34. 2021AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognitioncited by 36
  35. 2023Connecting Multi-modal Contrastive Representationscited by 36
  36. 2021Move2Hear: Active Audio-Visual Source Separationcited by 35
  37. 2021Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognitioncited by 35
  38. 2021The Right to Talk: An Audio-Visual Transformer Approachcited by 33

50 most cited of 259 All 259 (JSON)