mlsanityData to 7 Oct 2026

VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformer

arXiv 2203.04099 ↗First posted 8 Mar 2022cs.SD cs.CV cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

This paper presents an audio-visual approach for voice separation which produces state-of-the-art results at a low latency in two scenarios: speech and singing voice. The model is based on a two-stage network. Motion cues are obtained with a lightweight graph convolutional network that processes face landmarks. Then, both audio and motion features are fed to an audio-visual transformer which produces a fairly good estimation of the isolated target source. In a second stage, the predominant voice is enhanced with an audio-only network. We present different ablation studies and comparison to state-of-the-art methods. Finally, we explore the transferability of models trained for speech separation in the task of singing voice separation. The demos, code, and weights are available in https://ipcv.github.io/VoViT/

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 8 Mar 2022.

Builds on

29 of 43 references.

References · 29

  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  3. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  4. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  5. 2018The Sound of Pixelscited by 258
  6. 2020Towards Fast, Accurate and Stable 3D Dense Face Alignmentcited by 219
  7. 2019The Sound of Motionscited by 145
  8. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  9. 2019Co-Separating Sounds of Visual Objectscited by 110
  10. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 107
  11. 2020An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separationcited by 81
  12. 2021Audio-Visual Synchronisation in the wildcited by 39
17 more
  1. 2017Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networkscited by 36
  2. 2021The Right to Talk: An Audio-Visual Transformer Approachcited by 33
  3. 2019Time Domain Audio Visual Speech Separationcited by 30
  4. 2020FaceFilter: Audio-visual speech separation using still imagescited by 25
  5. 2020Multi-modal Multi-channel Target Speech Separationcited by 20
  6. 2020Conditioned Source Separation for Music Instrument Performancescited by 18
  7. 2018Face Landmark-based Speaker-Independent Audio-Visual Speech Enhancement in Multi-Talker Environmentscited by 12
  8. 2020Lite Audio-Visual Speech Enhancementcited by 12
  9. 2021A cappella: Audio-visual Singing Voice Separationcited by 12
  10. 2019Mixture of Inference Networks for VAE-based Audio-visual Speech Enhancementcited by 11
  11. 2021Multimodal Attention Fusion for Target Speaker Extractioncited by 10
  12. 2019WildMix Dataset and Spectro-Temporal Transformer Model for Monoaural Audio Source Separationcited by 7
  13. 2018On Training Targets and Objective Functions for Deep-Learning-Based Audio-Visual Speech Enhancementcited by 5
  14. 2021Improving On-Screen Sound Separation for Open-Domain Videos with Audio-Visual Self-Attentioncited by 5
  15. 2021VSEGAN: Visual Speech Enhancement Generative Adversarial Networkcited by 3
  16. 2021Audio-Visual Speech Separation Using Cross-Modal Correspondence Losscited by 2
  17. 2016Multi-Modal Hybrid Deep Neural Network for Speech Enhancementcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 3; 2025: 3; 2026: 1.

14 papers.

Sort:

By area:

  1. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  2. 2024Multimodal Action Quality Assessmentcited by 20
  3. 2022VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voicescited by 18
  4. 2024On the Audio Hallucinations in Large Audio-Video Language Modelscited by 15
  5. 2023IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separationcited by 11
  6. 2023LAVSS: Location-Guided Audio-Visual Spatial Audio Separationcited by 8
  7. 2023Speech inpainting: Context-based speech synthesis guided by videocited by 4
  8. 2023Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separationcited by 4
  9. 2024RT-LA-VocE: Real-Time Low-SNR Audio-Visual Speech Enhancementcited by 4
  10. 20233M-TRANSFORMER: A Multi-Stage Multi-Stream Multimodal Transformer for Embodied Turn-Taking Predictioncited by 2
  11. 2025TEn-CATG:Text-Enriched Audio-Visual Video Parsing with Multi-Scale Category-Aware Temporal Graphcited by 1
  12. 2025Seeing Soundscapes: Audio-Visual Generation and Separation from Soundscapes Using Audio-Visual Separatorcited by 0
2 more
  1. 2025Learning from Silence and Noise for Visual Sound Source Localizationcited by 0
  2. 2026MambaVoice: Lightweight Audiovisual Singing Voice Separation Via A Hybrid Mamba-Transformer Modelcited by 0