mlsanityData to 7 Oct 2026

The Sound of Motions

arXiv 1904.05979 ↗First posted 11 Apr 2019cs.CV cs.SD eess.AScited by 145

Not on the map. Why?

Abstract (arXiv)

Sounds originate from object motions and vibrations of surrounding air. Inspired by the fact that humans is capable of interpreting sound sources from how objects move visually, we propose a novel system that explicitly captures such motion cues for the task of sound localization and separation. Our system is composed of an end-to-end learnable model called Deep Dense Trajectory (DDT), and a curriculum learning scheme. It exploits the inherent coherence of audio-visual signals from a large quantities of unlabeled videos. Quantitative and qualitative evaluations show that comparing to previous models that rely on visual appearance cues, our motion based system improves performance in separating musical instrument sounds. Furthermore, it separates sound components from duets of the same category of instruments, a challenging problem that has not been addressed before.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 42; 2022: 42; 2023: 26; 2024: 20; 2025: 14; 2026: 1.

145 papers.

Sort:

By area:

  1. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  2. 2021Localizing Visual Sounds the Hard Waycited by 169
  3. 2022Audio-Visual Segmentationcited by 148
  4. 2021Taming Visually Guided Sound Generationcited by 136
  5. 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 115
  6. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  7. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
  8. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 93
  9. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  10. 2022Localizing Visual Sounds the Easy Waycited by 81
  11. 2021Robust Audio-Visual Instance Discriminationcited by 80
  12. 2023Audio-Visual Segmentation with Semanticscited by 73
38 more
  1. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 72
  2. 2022Learning Neural Acoustic Fieldscited by 65
  3. 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
  4. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  5. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  6. 2023Audio-Visual Grouping Network for Sound Localization from Mixturescited by 52
  7. 2022Contrastive Positive Sample Propagation along the Audio-Visual Event Linecited by 50
  8. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  9. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
  10. 2022Visual Acoustic Matchingcited by 45
  11. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  12. 2023Annotation-free Audio-Visual Segmentationcited by 44
  13. 2022Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisationcited by 43
  14. 2021MM-Pyramid: Multimodal Pyramid Attentional Network for Audio-Visual Event Localization and Video Parsingcited by 39
  15. 2021Audio-Visual Synchronisation in the wildcited by 39
  16. 2023STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Eventscited by 39
  17. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 38
  18. 2022Quantized GAN for Complex Music Generation from Dance Videoscited by 37
  19. 2021The Right to Talk: An Audio-Visual Transformer Approachcited by 33
  20. 2021Class-aware Sounding Objects Localization via Audiovisual Correspondencecited by 32
  21. 2022Learning Sound Localization Better From Semantically Similar Samplescited by 32
  22. 2022Disentangling Architecture and Training for Optical Flowcited by 32
  23. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  24. 2021MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modelingcited by 31
  25. 2023Sound Source Localization is All about Cross-Modal Alignmentcited by 30
  26. 2021Self-supervised object detection from audio-visual correspondencecited by 29
  27. 2021Visual Scene Graphs for Audio Source Separationcited by 29
  28. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  29. 2021Unsupervised Sound Localization via Iterative Contrastive Learningcited by 28
  30. 2023Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answeringcited by 28
  31. 2024Text-to-Audio Generation Synchronized with Videoscited by 28
  32. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  33. 2022Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Languagecited by 27
  34. 2023Discovering Sounding Objects by Audio Queries for Audio Visual Segmentationcited by 27
  35. 2024Audio-Visual Segmentation via Unlabeled Frame Exploitationcited by 27
  36. 2021Learning Audio-Visual Dereverberationcited by 26
  37. 2022WebUAV-3M: A Benchmark for Unveiling the Power of Million-Scale Deep UAV Trackingcited by 26
  38. 2021Beyond Image to Depth: Improving Depth Prediction using Echoescited by 24

50 most cited of 145 All 145 (JSON)