mlsanityData to 7 Oct 2026

Audio-Visual Scene Analysis with Self-Supervised Multisensory Features

arXiv 1804.03641 ↗First posted 10 Apr 2018cs.CV cs.SD eess.AScited by 335

Not on the map. Why?

Abstract (arXiv)

The thud of a bouncing ball, the onset of speech as lips open -- when visual and audio events occur together, it suggests that there might be a common, underlying event that produced both signals. In this paper, we argue that the visual and audio components of a video signal should be modeled jointly using a fused multisensory representation. We propose to learn such a representation in a self-supervised way, by training a neural network to predict whether video frames and audio are temporally aligned. We use this learned representation for three applications: (a) sound source localization, i.e. visualizing the source of sound in a video; (b) audio-visual action recognition; and (c) on/off-screen audio source separation, e.g. removing the off-screen translator's voice from a foreign official's speech. Code, models, and video results are available on our webpage: http://andrewowens.com/multisensory

Key facts. Cited by 335 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 92; 2022: 92; 2023: 66; 2024: 40; 2025: 29; 2026: 16.

335 papers.

Sort:

By area:

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  3. 2022Multimodal Learning with Transformers: A Surveycited by 399
  4. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  5. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  6. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  7. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  8. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  9. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  10. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  11. 2021Localizing Visual Sounds the Hard Waycited by 169
  12. 2022Audio-Visual Segmentationcited by 148
38 more
  1. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 147
  2. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  3. 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 125
  4. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  5. 2022PMR: Prototypical Modal Rebalance for Multimodal Learningcited by 114
  6. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  7. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  8. 2024Synchformer: Efficient Synchronization from Sparse Cuescited by 94
  9. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 93
  10. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  11. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  12. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 84
  13. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 84
  14. 2022Localizing Visual Sounds the Easy Waycited by 81
  15. 2021Robust Audio-Visual Instance Discriminationcited by 80
  16. 2022Object discovery and representation networkscited by 80
  17. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 73
  18. 2023Audio-Visual Segmentation with Semanticscited by 73
  19. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 72
  20. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  21. 2023Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lecturescited by 68
  22. 2022Audio Self-supervised Learning: A Surveycited by 65
  23. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  24. 2022See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulationcited by 61
  25. 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
  26. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  27. 2022MAViL: Masked Audio-Video Learnerscited by 56
  28. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  29. 2021Distilling Audio-Visual Knowledge by Compositional Contrastive Learningcited by 53
  30. 2021Home Action Genome: Cooperative Compositional Action Understandingcited by 52
  31. 2022Semi-Supervised and Unsupervised Deep Visual Learning: A Surveycited by 52
  32. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  33. 2024POP-3D: Open-Vocabulary 3D Occupancy Prediction from Imagescited by 52
  34. 2023AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesiscited by 49
  35. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  36. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  37. 2022Audiovisual Masked Autoencoderscited by 47
  38. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46

50 most cited of 335 All 335 (JSON)