mlsanityData to 7 Oct 2026

AV-CrossNet: an Audiovisual Complex Spectral Mapping Network for Speech Separation By Leveraging Narrow- and Cross-Band Modeling

arXiv 2406.11619 ↗First posted 17 Jun 2024eess.AS cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Adding visual cues to audio-based speech separation can improve separation performance. This paper introduces AV-CrossNet, an audiovisual (AV) system for speech enhancement, target speaker extraction, and multi-talker speaker separation. AV-CrossNet is extended from the CrossNet architecture, which is a recently proposed network that performs complex spectral mapping for speech separation by leveraging global attention and positional encoding. To effectively utilize visual cues, the proposed system incorporates pre-extracted visual embeddings and employs a visual encoder comprising temporal convolutional layers. Audio and visual features are fused in an early fusion layer before feeding to AV-CrossNet blocks. We evaluate AV-CrossNet on multiple datasets, including LRS, VoxCeleb, and COG-MHEAR challenge. Evaluation results demonstrate that AV-CrossNet advances the state-of-the-art performance in all audiovisual tasks, even on untrained and mismatched datasets.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 42 of its 64 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

42 of 64 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  3. 2018SDR - half-baked or well done?cited by 420
  4. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  5. 2018Deep Audio-Visual Speech Recognitioncited by 318
  6. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  7. 2017Supervised Speech Separation Based on Deep Learning: An Overviewcited by 213
  8. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 211
  9. 2016Temporal Convolutional Networks: A Unified Approach to Action Segmentationcited by 199
  10. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  11. 2019Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separationcited by 182
  12. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 160
30 more
  1. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 154
  2. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  3. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 113
  4. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 107
  5. 2017Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networkscited by 97
  6. 2020An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separationcited by 81
  7. 2018VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Maskingcited by 72
  8. 2022TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separationcited by 62
  9. 2023ICASSP 2023 Deep Noise Suppression Challengecited by 62
  10. 2021Speech Separation Using an Asynchronous Fully Recurrent Convolutional Neural Networkcited by 32
  11. 2019Time Domain Audio Visual Speech Separationcited by 30
  12. 2021Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separationcited by 27
  13. 2019My lips are concealed: Audio-visual speech enhancement through obstructionscited by 26
  14. 2020FaceFilter: Audio-visual speech separation using still imagescited by 25
  15. 2020Muse: Multi-modal target speaker extraction with visual cuescited by 21
  16. 2019Divide and Conquer: A Deep CASA Approach to Talker-independent Monaural Speaker Separationcited by 20
  17. 2023SpatialNet: Extensively Learning Spatial Information for Multichannel Joint Speech Separation, Denoising and Dereverberationcited by 17
  18. 2023AV-SepFormer: Cross-Attention SepFormer for Audio-Visual Target Speaker Extractioncited by 15
  19. 2019Audio-Visual Speech Separation and Dereverberation with a Two-Stage Multimodal Networkcited by 13
  20. 2020Lite Audio-Visual Speech Enhancementcited by 12
  21. 2019Mixture of Inference Networks for VAE-based Audio-visual Speech Enhancementcited by 11
  22. 2021Many-Speakers Single Channel Speech Separation with Optimal Permutation Trainingcited by 11
  23. 2023IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separationcited by 11
  24. 2022An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuitscited by 9
  25. 2022A Hybrid Continuity Loss to Reduce Over-Suppression for Time-domain Target Speaker Extractioncited by 7
  26. 2024Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extractioncited by 7
  27. 2023Scenario-Aware Audio-Visual TF-GridNet for Target Speech Extractioncited by 5
  28. 2023Audio-Visual Speech Separation in Noisy Environments with a Lightweight Iterative Modelcited by 4
  29. 2023Multi-channel Conversational Speaker Separation via Neural Diarizationcited by 2
  30. 2024CrossNet: Leveraging Global, Cross-Band, Narrow-Band, and Positional Encoding for Single- and Multi-Channel Speaker Separationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026LLM-Guided Reinforcement Learning for Audio-Visual Speech Enhancementcited by 0