mlsanityData to 14 Oct 2026

Audio-Visual Speech Separation in Noisy Environments with a Lightweight Iterative Model

arXiv 2306.00160 ↗First posted 31 May 2023eess.AS cs.LG cs.SDcited by 4

Not on the map. Why?

Abstract (arXiv)

We propose Audio-Visual Lightweight ITerative model (AVLIT), an effective and lightweight neural network that uses Progressive Learning (PL) to perform audio-visual speech separation in noisy environments. To this end, we adopt the Asynchronous Fully Recurrent Convolutional Neural Network (A-FRCNN), which has shown successful results in audio-only speech separation. Our architecture consists of an audio branch and a video branch, with iterative A-FRCNN blocks sharing weights for each modality. We evaluated our model in a controlled environment using the NTCD-TIMIT dataset and in-the-wild using a synthetic dataset that combines LRS3 and WHAM!. The experiments demonstrate the superiority of our model in both settings with respect to various audio-only and audio-visual baselines. Furthermore, the reduced footprint of our model makes it suitable for low resource applications.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

20 of 36 references.

References · 20

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2018SDR - half-baked or well done?cited by 423
  3. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 406
  4. 2018Deep Audio-Visual Speech Recognitioncited by 322
  5. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  6. 2019Progressive Image Deraining Networks: A Better and Simpler Baselinecited by 272
  7. 2017Supervised Speech Separation Based on Deep Learning: An Overviewcited by 216
  8. 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 195
  9. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 191
  10. 2020Attention is All You Need in Speech Separationcited by 185
  11. 2019Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separationcited by 182
  12. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 156
8 more
  1. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  2. 2020An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separationcited by 81
  3. 2020Deep Audio-Visual Learning: A Surveycited by 69
  4. 2020Towards Practical Lipreading with Distilled and Efficient Modelscited by 45
  5. 2022An efficient encoder-decoder architecture with top-down attention for speech separationcited by 38
  6. 2021Speech Separation Using an Asynchronous Fully Recurrent Convolutional Neural Networkcited by 32
  7. 2019Time Domain Audio Visual Speech Separationcited by 30
  8. 2020Lite Audio-Visual Speech Enhancementcited by 12

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1; 2025: 1.

4 papers.

  1. 2023IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separationcited by 11
  2. 2023RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separationcited by 2
  3. 2025Efficient Audio-Visual Speech Separation with Discrete Lip Semantics and Multi-Scale Global-Local Attentioncited by 2
  4. 2024AV-CrossNet: an Audiovisual Complex Spectral Mapping Network for Speech Separation By Leveraging Narrow- and Cross-Band Modelingcited by 1