mlsanityData to 7 Oct 2026

VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency

arXiv 2101.03149 ↗First posted 8 Jan 2021cs.CV cs.SD eess.IVcited by 114

Not on the map. Why?

Abstract (arXiv)

We introduce a new approach for audio-visual speech separation. Given a video, the goal is to extract the speech associated with a face in spite of simultaneous background sounds and/or other human speakers. Whereas existing methods focus on learning the alignment between the speaker's lip movements and the sounds they generate, we propose to leverage the speaker's face appearance as an additional prior to isolate the corresponding vocal qualities they are likely to produce. Our approach jointly learns audio-visual speech separation and cross-modal speaker embeddings from unlabeled video. It yields state-of-the-art results on five benchmark datasets for audio-visual speech separation and enhancement, and generalizes well to challenging real-world videos of diverse scenarios. Our video results and code: http://vision.cs.utexas.edu/projects/VisualVoice/.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Builds on 53 of its 86 references that are in the map. First posted on arXiv 8 Jan 2021.

Builds on

53 of 86 references.

References · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  4. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  5. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  6. 2017Look, Listen and Learncited by 452
  7. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  8. 2018Deep Audio-Visual Speech Recognitioncited by 318
  9. 2018The Sound of Pixelscited by 258
  10. 2017Objects that Soundcited by 254
  11. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  12. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
38 more
  1. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  2. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 211
  3. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  4. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  5. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  6. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  7. 2019The Sound of Motionscited by 145
  8. 2017S3FD: Single Shot Scale-invariant Face Detectorcited by 142
  9. 2015Visually Indicated Soundscited by 128
  10. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  11. 2020Music Gesture for Visual Sound Separationcited by 112
  12. 2019Co-Separating Sounds of Visual Objectscited by 110
  13. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 107
  14. 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
  15. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  16. 2020Lipreading using Temporal Convolutional Networkscited by 98
  17. 2017Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networkscited by 97
  18. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  19. 2020Spot the conversation: speaker diarisation in the wildcited by 75
  20. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  21. 20182.5D Visual Soundcited by 63
  22. 2018Seeing Voices and Hearing Faces: Cross-modal biometric matchingcited by 60
  23. 2018Learnable PINs: Cross-Modal Embeddings for Person Identitycited by 60
  24. 2018Perfect match: Improved cross-modal embeddings for audio-visual synchronisationcited by 60
  25. 2019Speech2Face: Learning the Face Behind a Voicecited by 55
  26. 2020Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separationcited by 55
  27. 2020Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Soundscited by 55
  28. 2020VisualEchoes: Spatial Image Representation Learning through Echolocationcited by 54
  29. 2019Recursive Visual Sound Separation Using Minus-Plus Netcited by 53
  30. 2018Emotion Recognition in Speech using Cross-Modal Transfer in the Wildcited by 48
  31. 2020Voice Separation with an Unknown Number of Multiple Speakerscited by 46
  32. 2019Self-Supervised Audio-Visual Co-Segmentationcited by 44
  33. 2020Towards Practical Lipreading with Distilled and Efficient Modelscited by 44
  34. 2017Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networkscited by 36
  35. 2020Disentangled Speech Embeddings using Cross-modal Self-supervisioncited by 30
  36. 2018On Learning Associations of Faces and Voicescited by 26
  37. 2019My lips are concealed: Audio-visual speech enhancement through obstructionscited by 26
  38. 2020FaceFilter: Audio-visual speech separation using still imagescited by 25

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 31; 2023: 30; 2024: 19; 2025: 14; 2026: 10.

114 papers.

Sort:

By area:

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  3. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  4. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  5. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 94
  6. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 84
  7. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 72
  8. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  9. 2022SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memorycited by 65
  10. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  11. 2022Contrastive Positive Sample Propagation along the Audio-Visual Event Linecited by 50
  12. 2022Visual Acoustic Matchingcited by 45
38 more
  1. 2023Annotation-free Audio-Visual Segmentationcited by 44
  2. 2022Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisationcited by 43
  3. 2023Separate Anything You Describecited by 43
  4. 2021Move2Hear: Active Audio-Visual Source Separationcited by 35
  5. 2022Learning Sound Localization Better From Semantically Similar Samplescited by 32
  6. 2023Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Taskscited by 31
  7. 2023Sound Source Localization is All about Cross-Modal Alignmentcited by 30
  8. 2022Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsingcited by 29
  9. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  10. 2023Progressive Spatio-temporal Perception for Audio-Visual Question Answeringcited by 29
  11. 2021Unsupervised Sound Localization via Iterative Contrastive Learningcited by 28
  12. 2022Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localizationcited by 27
  13. 2023Imaginary Voice: Face-styled Diffusion Model for Text-to-Speechcited by 27
  14. 2024Audio-Visual Segmentation via Unlabeled Frame Exploitationcited by 27
  15. 2021Learning Audio-Visual Dereverberationcited by 26
  16. 2021AVA-AVD: Audio-Visual Speaker Diarization in the Wildcited by 24
  17. 2022Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesiscited by 24
  18. 2022AudioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separationcited by 24
  19. 2024FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matchingcited by 23
  20. 2023AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generationcited by 22
  21. 2023Class-Incremental Grouping Network for Continual Audio-Visual Learningcited by 22
  22. 2022Look&Listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancementcited by 20
  23. 2023Weakly-Supervised Audio-Visual Segmentationcited by 19
  24. 2022Sound Localization by Self-Supervised Time Delay Estimationcited by 18
  25. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  26. 2021Structure from Silence: Learning Scene Structure from Ambient Soundcited by 17
  27. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  28. 2022Active Audio-Visual Separation of Dynamic Sound Sourcescited by 16
  29. 2022MarginNCE: Robust Sound Localization with a Negative Margincited by 16
  30. 2023CASP-Net: Rethinking Video Saliency Prediction from an Audio-VisualConsistency Perceptual Perspectivecited by 15
  31. 2022VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformercited by 14
  32. 2023Revisit Weakly-Supervised Audio-Visual Video Parsing from the Language Perspectivecited by 14
  33. 2021A cappella: Audio-visual Singing Voice Separationcited by 12
  34. 2023Audio-Visual Speech Enhancement with Score-Based Generative Modelscited by 12
  35. 2024Modeling and Driving Human Body Soundfields through Acoustic Primitivescited by 12
  36. 2023Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised Audio-Visual Video Parsingcited by 11
  37. 2023IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separationcited by 11
  38. 2024SoundingActions: Learning How Actions Sound from Narrated Egocentric Videoscited by 11

50 most cited of 114 All 114 (JSON)