mlsanityData to 7 Oct 2026

V-SlowFast Network for Efficient Visual Sound Separation

arXiv 2109.08867 ↗First posted 18 Sep 2021cs.CV cs.SD eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

The objective of this paper is to perform visual sound separation: i) we study visual sound separation on spectrograms of different temporal resolutions; ii) we propose a new light yet efficient three-stream framework V-SlowFast that operates on Visual frame, Slow spectrogram, and Fast spectrogram. The Slow spectrogram captures the coarse temporal resolution while the Fast spectrogram contains the fine-grained temporal resolution; iii) we introduce two contrastive objectives to encourage the network to learn discriminative visual features for separating sounds; iv) we propose an audio-visual global attention module for audio and visual feature fusion; v) the introduced V-SlowFast model outperforms previous state-of-the-art in single-frame based visual sound separation on small- and large-scale datasets: MUSIC-21, AVE, and VGG-Sound. We also propose a small V-SlowFast architecture variant, which achieves 74.2% reduction in the number of model parameters and 81.4% reduction in GMACs compared to the previous multi-stage models. Project page: https://ly-zhu.github.io/V-SlowFast

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 47 of its 67 references that are in the map. First posted on arXiv 18 Sep 2021.

Builds on

47 of 67 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  8. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  9. 2018SlowFast Networks for Video Recognitioncited by 1,862
  10. 2019Contrastive Multiview Codingcited by 1,297
  11. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  12. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
35 more
  1. 2017Look, Listen and Learncited by 452
  2. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  3. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  4. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  5. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  6. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  7. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  8. 2018The Sound of Pixelscited by 258
  9. 2017Objects that Soundcited by 254
  10. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  11. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  12. 2020Self-supervised Co-training for Video Representation Learningcited by 232
  13. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  14. 2021Localizing Visual Sounds the Hard Waycited by 169
  15. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  16. 2019The Sound of Motionscited by 145
  17. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  18. 2019SCSampler: Sampling Salient Clips from Video for Efficient Action Recognitioncited by 133
  19. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  20. 2020Music Gesture for Visual Sound Separationcited by 112
  21. 2019Co-Separating Sounds of Visual Objectscited by 110
  22. 2020Learning Representations from Audio-Visual Spatial Alignmentcited by 95
  23. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  24. 2020Video Representation Learning by Recognizing Temporal Transformationscited by 82
  25. 2020Foley Music: Learning to Generate Music from Videoscited by 82
  26. 2020Semantic Audio-Visual Navigationcited by 79
  27. 2020Evolving Losses for Unsupervised Video Representation Learningcited by 74
  28. 2020Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learningcited by 63
  29. 2021Slow-Fast Auditory Streams For Audio Recognitioncited by 56
  30. 2020VisualEchoes: Spatial Image Representation Learning through Echolocationcited by 54
  31. 2019Recursive Visual Sound Separation Using Minus-Plus Netcited by 53
  32. 2020Telling Left from Right: Learning Spatial Correspondence of Sight and Soundcited by 43
  33. 2021Visually Guided Sound Source Separation and Localization using Self-Supervised Motion Representationscited by 18
  34. 2020Visually Guided Sound Source Separation using Cascaded Opponent Filter Networkcited by 15
  35. 2019Listen to the Imagecited by 3

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 0; 2024: 2; 2025: 1.

6 papers.

  1. 2021Audio-Visual Synchronisation in the wildcited by 39
  2. 2022Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Languagecited by 27
  3. 2025VGGSounder: Audio-Visual Evaluations for Foundation Modelscited by 8
  4. 2022Beyond Visual Field of View: Perceiving 3D Environment with Echoes and Visioncited by 4
  5. 2024Anchors Aweigh! Sail for Optimal Unified Multi-Modal Representationscited by 4
  6. 2024Modulating State Space Model with SlowFast Framework for Compute-Efficient Ultra Low-Latency Speech Enhancementcited by 4